Uncovering how a stream of sensory inputs maps onto a sequence of actions is crucial for understanding the nervous system. However, behavior is shaped not only by recent sensory stimuli but also by longer-term experiences, internal neuronal dynamics, and physiological states. As a result, the same experimental conditions can result in variable behaviors. We study the role of internal states in zebrafish larvae behavior, leveraging a rich dataset of three million movements recorded during spontaneous behavior, photomotor responses, optomotor responses, and visually driven escape in the zebrafish larvae. We employ an HMM-GLM model, that combines sensory-motor mapping via generalized linear models (GLM) with hidden Markov models (HMM) and identify a series of internal states falling into three main categories: passive, cruising, and wandering. Our results show that passive states evolve with dynamics extending several minutes beyond stimulus presentation. Predictions of the response to external stimuli or pharmacological manipulations are improved when accounting for internal states, validating our approach.