We have studied the small scale distribution of atomic hydrogen (HI) using 21-cm absorption spectra against multiple-component background radio continuum sources from the 21-SPONGE survey and the Millennium Arecibo Absorption Line Survey. We have found $>5\sigma$ optical depth variations at a level of $\sim0.03-0.5$ between 13 out of 14 adjacent sightlines separated by a few arcseconds to a few arcminutes, suggesting the presence of neutral structures on spatial scales from a few to thousands of AU (which we refer to as tiny scale atomic structure, TSAS)... The optical depth variations are strongest in directions where the HI column density and the fraction of HI in the cold neutral medium (CNM) are highest, which tend to be at low Galactic latitudes. By measuring changes in the properties of Gaussian components fitted to the absorption spectra, we find that changes in both the peak optical depth and the linewidth of TSAS absorption features contribute to the observed optical depth variations, while changes in the central velocity do not appear to strongly impact the observed variations. Both thermal and turbulent motions contribute appreciably to the linewidths, but the turbulence does not appear strong enough to confine overpressured TSAS. In a majority of cases, the TSAS column densities are sufficiently high that these structures can radiatively cool fast enough to maintain thermal equilibrium with their surroundings, even if they are overpressured. We also find that a majority of TSAS is associated with the CNM. For TSAS in the direction of the Taurus molecular cloud and the local Leo cold cloud, we estimate densities over an order of magnitude higher than typical CNM densities. (read more)