The auditory nerve encodes sound with spike-timing that is phase locked to the fine-grained temporal structure of sound. The precision of this coding substantially exceeds that of any other sensory modality, but its role in hearing remains controversial because physiological mechanisms for extracting information from the spike timing have proven elusive. We investigated the perceptual role of auditory nerve phase locking using deep artificial neural networks in the spirit of ideal observer analysis. Neural networks took input from a simulated cochlea and were optimized for natural tasks. We examined whether phase locking in a network’s cochlear input was necessary to obtain human-like behavior. We trained networks to recognize and localize words, voices, and environmental sounds using simulated auditory nerve representations of naturalistic auditory scenes. We manipulated the upper limit of phase locking via the lowpass cutoff in simulated inner hair cells. Networks using high-frequency phase locking replicated key aspects of human auditory behavior: task performance was robust to sound level and background noise. Degrading phase locking impaired performance, but much more so on some tasks than others, substantially impairing voice recognition and sound localization while leaving word recognition largely intact. The results suggest that auditory nerve phase locking is critical for accurate sound localization, for voice recognition in noise, and for level-invariant hearing. These findings may clarify conditions in which prostheses that fail to restore high-fidelity temporal coding (e.g., contemporary cochlear implants) should and should not be expected to restore near-normal hearing.