Uncertainty is a fundamental aspect of the natural environment. Computing with uncertain information is important not only for encoding and decoding stimuli in early sensory areas, but also for more complex downstream inference and integration. The head-direction (HD) system is a prominent example of this, where noisily encoded angular velocity estimates need to be inferred and integrated into a circular attractor to keep track of the heading direction. The brain may employ sampling-based methods to perform inference, where neural activity reflects samples from the posterior probability distribution. However, how sampling-based methods might be linked with the attractor dynamics of HD computations is poorly understood. Here, we derive a spiking neural network with local nonlin- earities that performs sampling-based inference and integration of noisily encoded inputs. To this end, we extend the sampling capabilities of a class of spiking networks --- previously constrained to sampling from Gaussian distributions --- to the exponential family distributions. We describe the precise requirements at the network, cellular, and synaptic levels for sampling from distributions with different moment functions, and implement sampling-based inference of latent stimuli encoded by noisy Poisson neurons. We then use this to develop a model for the HD system, in which inferred angular velocities are integrated on a ring-shaped prior. The model bridges sampling-based inference with attractor dynamics and provides concrete, testable predictions about correlated subthreshold voltages, short- and long-term neural activity correlation patterns, and statistics of bump movement. Our work showcases how sampling-based computations could be used in higher-order brain areas, paving the road for bridging probabilistic methods with attractor dynamics in spiking neural networks. Moreover, we offer an alternative perspective on the neural computations responsible for orientation and navigation, providing testable predictions about neural activity that can be examined in future neuroscientific experiments.