The activity of 100s-1000’s of neurophysiological signals can be recorded during behaviors and in response to sensory stimuli, creating high-dimensional time-series data. Understanding such data is often aided by methods that extract low-dimensional latent structure present in the high-dimensional recordings. Multi-output Gaussian process models leverage the nonparametric nature of Gaussian processes to capture structure across multiple outputs. However, this class of models typically assumes that the correlations between the output response variables are invariant in the input space (e.g., sensory stimuli). We present the stochastic linear mixing model (SLMM), which utilizes a conditional linear mapping function between latent variables and observations without loss of the geometric interpretation of the subspace. In our formulation, the mixture coefficients depend on inputs, making SLMM flexible and effective to capture complex output dependencies. However, the inference for SLMMs is intractable for large datasets, making them inapplicable to modern neuroscience data. Thus, we propose a new regression framework, the orthogonal stochastic linear mixing model (OSLMM), that introduces an orthogonality constraint amongst the stochastic mixing coefficients. This constraint dramatically reduces the computational burden of inference while retaining the capability to handle complex output dependencies, and also contributes to extraction of more interpretable latent trajectories. Moreover, we put a uniform prior on the orthogonal subspace (Stiefel manifold) to manage the variability of subspaces. We provide Markov chain Monte Carlo (MCMC) inference procedures for both SLMM and OSLMM, and demonstrate superior model scalability and reduced prediction error of OSLMM compared with state-of-the-art methods in several real-world data-sets. In neurophysiology recordings from auditory cortex, we use the inferred latent functions for compact visualization of population responses to stimuli, and demonstrate superior results compared to a competing method (GPFA). Together, these results demonstrate that OSLMM will be useful for analysis of large-scale time-series data increasingly common in neuroscience.