Diversity in the natural world emerges from the collective behavior of large numbers of interacting objects. Statistical physics provides the framework relating microscopic to macroscopic properties. A fundamental assumption underlying this approach is that we have complete knowledge of the interactions between the microscopic entities. But what if that, even though possible in principle becomes impossible in practice ? Can we still construct a framework for describing their collective behavior ? Dense suspensions and granular materials are two often quoted examples where we face this challenge. These are systems where because of the complicated surface properties of particles there is extreme sensitivity of the interactions to particle positions. In this talk, I will present a perspective based on notions of constraint satisfaction that provides a way forward. I will focus on our recent work on the emergence of elasticity in the absence of any broken symmetry, and sketch out other problems that can be addressed using this perspective.