Early quiescent galaxies at z~2 are known to be remarkably compact compared to their nearby counterparts. Possible progenitors of these systems include galaxies that are structurally similar, but are still rapidly forming stars... Here, we present Karl G. Jansky Very Large Array (VLA) observations of the CO(1-0) line towards three such compact, star-forming galaxies at z~2.3, significantly detecting one. The VLA observations indicate baryonic gas fractions >~5 times lower and gas depletion times >~10 times shorter than normal, extended massive star-forming galaxies at these redshifts. At their current star formation rates, all three objects will deplete their gas reservoirs within 100Myr. These objects are among the most gas-poor objects observed at z>2, and are outliers from standard gas scaling relations, a result which remains true regardless of assumptions about the CO-H2 conversion factor. Our observations are consistent with the idea that compact, star-forming galaxies are in a rapid state of transition to quiescence in tandem with the build-up of the z~2 quenched population. In the detected compact galaxy, we see no evidence of rotation or that the CO-emitting gas is spatially extended relative to the stellar light. This casts doubt on recent suggestions that the gas in these compact galaxies is rotating and significantly extended compared to the stars. Instead, we suggest that, at least for this object, the gas is centrally concentrated, and only traces a small fraction of the total galaxy dynamical mass. (read more)