Factorization of visual scenes refers to disentangling explanatory variables such that they are represented explicitly. For example, to act in the world, animals must continually distinguish between object form and motion. We propose a model of primary visual cortex (V1) consisting of complex cell-like units. Each unit has two sub-units and corresponding receptive fields (RFs), and factorizes its amplitude and phase response through a recurrent inference procedure. The model also learns localized, oriented, and band-pass RFs from natural images, and exhibits signature V1 complex cell phase invariance. Our primary findings are: 1) without motion cues, the model learns approximately quadrature-like Gabor filter pairs; 2) additionally, the model learns filter pairs that parameterize transformations other than classic local translation; 3) despite phase-invariant properties, the model exhibits increased selectivity compared to the standard model of complex cells, and 4) the structure of the inferred phase statistics exhibits clear dependencies, suggesting signals for subsequent computation.