Inferring the underlying connectivity of a network from observations of its activity is a long-standing challenge. In the brain, this challenge is exacerbated by dynamic reconfiguration of effective connectivity, mediated by both synaptic plasticity and neuromodulation. Traditional methods for extracting connectivity from activity focus on modeling static connectivity. On the other hand, recent transformer-based models are able to capture nonstationary interactions, but their deep layered structures and nonlinear attention mechanisms make them challenging to interpret, particularly in terms of connectivity among units in the original data. To bridge this gap, we propose a transformer-inspired network model, the NetFormer, for which the linearized core of the attention mechanism without the softmax activation directly encodes the nonstationary and nonlinear structure of networks. In NetFormer, the activity of each neuron across a series of historical time steps is defined as a token. These tokens are then linearly mapped through a query and key mechanism to generate a state- (and hence time-) dependent attention matrix that encodes nonstationary connectivity structures. We apply NetFormer to a large-scale dataset of real neural recordings, which contains neural activity, cell type, and behavioral state information. NetFormer effectively predicts neural dynamics and identifies cell-type specific, state-dependent dynamic connectivity that matches patterns measured in separate ground-truth physiology experiments, highlighting its potential in decoding in-vivo neuronal interactions based on functional activity observations. We further demonstrate NetFormer's ability to model a key feature of neuronal networks, the spike-timing-dependent plasticity, whereby connection strengths continually change in response to local activity patterns. We also provide a sketch of the mathematical intuition behind NetFormer from the perspective of nonstationary and nonlinear networked dynamical systems.