Accumulating evidence has substantiated that various brain functions are encoded by a sparse population of experience-activated neurons. However, it remains elusive whether stress-responsive neurons ever exist and contribute to the pathogenesis of depression. Utilizing activity-dependent viral strategies, here we identified sparse stress-responsive neurons primarily located in the middle part of the lateral hypothalamus (mLH) and the medial part of the lateral habenula (LHbM), which were universally recruited by various stressors. Remarkably, merely silencing these stress-responsive neurons entirely eliminated the development of depression-like state under chronic stress (CS), whereas photostimulation sufficiently incepted persistent depression-like state in mice without experiencing CS. Stress-responsive neurons formed dominant connections, primarily mediating the mLH-LHb excitation and selectively responsible for CS-altered circuit architecture. Intriguingly, stress-responsive neurons in LHb presumably serves as “starter” cells to locally innervate LHb and coordinate LHb hyperactivity under CS. Collectively, our study unveils the core functional unit within mLH-LHbM circuit in gating the pathogenesis of depression.