Neural circuits must perform probabilistic computations to efficiently process noisy sensory information. Sampling codes propose that single neuron variability is a signature of probabilistic computation, and corresponds to sampling the space of possible solutions in proportion to their posterior probability. Under this hypothesis, the neural code defines how variables inferred by the network are represented in single neurons. Compared to other proposed probabilistic codes, the capacity of sampling codes scales with the number of neurons, but their convergence speed scales poorly with dimensionality of the parameter space. To be useful to the organism, the proposed probabilistic computations should match perceptual speed. Work in statistics and machine learning shows that inference can be accelerated by sampling on a manifold with desirable geometry. However, these methods require structured noise, which in biological networks would imply strong electrical coupling across neurons. Here, we propose that the neural code can implement such favorable geometry in electrically uncoupled neurons, using the formalism of mirror descent. We first present a multivariate Gaussian model to highlight how distributed codes can implement a favorable geometry to achieve accelerated inference, independent of the dimensionality of the problem. Next, we apply these principles to neural circuits in the olfactory bulb, using a Poisson noise model for the activity of olfactory receptor neurons. Excitatory projection cells (mitral/tufted cells) implement a form of predictive coding, while inhibitory neurons (granule cells) implement sampling and control the geometry of representations. Since granule cells greatly outnumber mitral/tufted cells, they can implement such geometry through a sparse code. We show that this distributed code accelerates the inference and avoids interference by distractor odors. To conclude, choosing a neural code that implements a favorable geometry accelerates inference, and we map such an algorithm onto neural circuits in the early olfactory system.