Disentangling independent latent components from neural data is a desirable goal in neuroscience. Independent component analysis (ICA) has been used to identify disentangled latent neural trajectories and their corresponding sub-brain maps from recorded imaging sequences. However, enforcing full independence between all latent components may be overly restrictive, as each independent neural trajectory and sub-brain map results in a rank-1 imaging sequence. Additionally, the connectivity inferred from a rank-1 sub-brain map produces a rank-1 connectivity matrix, which may lack expressiveness. To address these limitations, we propose the Partially Disentangled VAE (PDisVAE)---a framework that can handle group-wise independence, allowing each independent latent group and its corresponding connectivity to exhibit a higher-than-one within-group rank. The PDisVAE model can naturally reduce to either the standard VAE or the fully disentangled VAE (ICA) by adjusting the number of groups and the within-group rank. We apply the PDisVAE to a dorsal voltage imaging dataset recorded from a mouse while receiving a unilateral air puff while learning a task. With assumptions of partial independence and higher within-group rank, the disentangled neural trajectories and their corresponding sub-brain maps are more interpretable and effectively reconstruct the original imaging sequences. Moreover, the rank-2 connectivity derived from these independent groups reveals more informative connections than those found in rank-1 connectivity.