While the desire to show evidence of progress is understandable, such pressures can encourage agencies to publish misleading results. The 2011 Los Angeles Homeless Count is a textbook case of manipulative statistical gymnastics.
In this week’s Los Angeles Times the Los Angeles Homeless Services Authority (LAHSA), the government organization charged with reducing homelessness throughout much of Los Angeles County, announced that homelessness had decreased by 3% county-wide from 2009 to 2011.
On face, the announcement of a 3% decrease in homelessness falls anywhere between good news and non-news. It is good news if you are culled into believing that the Homeless Count is actually a census (which it is not) and is non-news if you recognize homeless counts for what they really are, estimates.
Census, Estimates, and False Precision
A census is an enumeration of a population, such as the Decennial Census administered by the United States government. When conducting a census, you are supposed to count, rather than estimate, all individuals that make up a population.
Despite the misleading name “homeless count”, homeless counts are in fact conducted through a combination of counting and sampling. Indeed, the 2011 Los Angeles Homeless Count was a hodge-podge of reported counts from homeless shelters, call-center sampling, and a massive volunteer driven counting effort that placed trained enumerators in about half of the Census tracts throughout the County.
Let me be clear here, counting all the homeless persons throughout a large geography is difficult, probably impossible. The approach used in this report is actually not bad either. But it is also not a census.
The distinction between a census and an estimate is important because the homeless counts provide a false sense of precision, which leads to inaccurate inference.
The claim that the 2011 Los Angeles Homeless Count identified with certitude 51,340 homeless persons is preposterous. Because the count is an estimate, these numbers should be reported with error bounds. For example, the report might find that with a high degree of likelihood there there were between 45,000 and 55,000 homeless persons in the county over the reporting period.
This concept of a margin of error is critically important, and the absence of error bounds renders the data provided by the count practically unusable. Indeed, LAHSA is promoting a marginal decrease in homelessness as evidence of social impact when in fact it is only evidence of statistical error.
Up, Down, and All Around
With the notion of sampling and error margins in mind, let’s take a closer look at this 3% decrease in homelessness. To be able to say that such a small change from one period to the next is significant, you have to know your error margins are smaller than the increase or decrease you are reporting in population size you are reporting.
That means in order to actually say there was a 3% decrease, we have to know that the reported numbers of homeless persons is essentially dead on, which they aren’t, because they are estimates. Therefore, whether homelessness actually decreased is likely statistically unprovable, because the error margins of the estimated homeless population likely exceed the reported population decrease.
Of course, faced with the pressure to show success, the authors of the report failed to include any error margins at all, touting any decrease as both meaningful and statistically significant.
What’s worse is that this supposed decrease from 2009 to 2011 does not actually use the original 2009 findings at all. The 2011 report claims there were 52,931 homeless people counted in 2009, while the original 2009 report found just 48,053.
Still with me?
The 3% decrease reported in the 2011 report uses the updated 2009 count, which shows a greater number of people being homeless in 2009 than the original report. If instead we use the originally reported 2009 estimation of 48,053 then homelessness would have actually increased by 6.8% from 2009 rather than decreased.
So which is it, did homelessness increase or decrease from 2009 to 2011? LAHSA seems to claim both, when in fact we simply don’t know.
Minor variations from one count to another results in a statistical wash, meaning as best we can tell things probably stayed about the same from 2009 to 2011. Instead these minor variations are being paraded around as meaningful accomplishments, which they are not.
The fact that the authors of the report saw fit to not only fail to calculate error margins but also to amend past counts, thus manufacturing meaningless decreases, is frankly statistically dishonest.
Whether homelessness increased or decreased from 2009 to 2011 is something we do not know the answer to. This report provides confusion, rather than answers to that question. The only thing we do know with from this report is the directional pull it has on the collective statistical aptitude of the homeless services sector.
Photo credit: InertiaCreeps