Accumulating evidence has substantiated that various brain functions are encoded by a sparse ensemble of experience-activated neurons. However, it remains elusive whether stress-specific ensembles ever exist and contribute to the pathogenesis of depression. Utilizing activity-dependent viral strategies, here we identified sparse stress ensembles primarily located in the middle part of the lateral hypothalamus (mLH) and the medial part of the lateral habenula (LHbM), which were universally recruited by various stressors. Remarkably, merely silencing these sparse ensembles entirely eliminated the development of depression-like state during chronic stress (CS), whereas photostimulation sufficiently incepted depression-like state without CS. Stress ensembles formed dominant connections, primarily mediating the mLH-LHb excitation and selectively responsible for CS-altered circuit architecture. Intriguingly, LHb stress ensemble presumably serves as “starter” cells to locally innervate LHb and coordinate LHb hyperactivity under CS. Collectively, these findings suggest that a sparse mLH- LHbM circuit predominantly controls stress encoding and the pathogenesis of depression.