One of the most robust observations of the stellar initial mass function (IMF) is its near-universality in the Milky Way and neighboring galaxies. But recent observations of early-type galaxies can be interpreted to imply a bottom-heavy IMF, while others of ultra-faint dwarfs could imply a top-heavy IMF... This would impose powerful constraints on star formation models. We explore what sort of cloud-scale IMF models could possibly satisfy these constraints. We utilize simulated galaxies which reproduce (broadly) the observed galaxy properties, while they also provide the detailed star formation history and properties of each progenitor star-forming cloud. We then consider generic models where the characteristic mass of the IMF is some arbitrary power-law function of progenitor cloud properties, along with well-known literature IMF models, which scale with Jeans mass, turbulent Bonnor-Ebert mass, temperature, the opacity limit, metallicity, or the protostellar heating mass. We show that no IMF models currently in the literature - nor any model where the turnover mass is an arbitrary power-law function of a combination of cloud temperature/density/size/metallicity/velocity dispersion/magnetic field - can reproduce the claimed IMF variation in ellipticals or dwarfs without severely violating observational constraints in the Milky Way. Specifically, they predict too much variation in the extreme environments of the Galaxy, compared to that observed. Either the IMF varies in a more complicated manner, or alternative interpretations of the extragalactic observations must be explored. (read more)