When Testing Hurts: Why the New York City Department of Homeless Services is Wrong

By | Oct 19, 2010

I’ve written before that program evaluation is all-important, allowing us as social sector practitioners to determine what kinds of poverty alleviation programs work and what kinds don’t. But, as David Henderson wrote last week, there is also a dark side to experimenting with social policy: we can hurt the people we are trying to help.

The example David wrote about is the New York City Department of Homeless Services’s decision to evaluate their HomeBase program by randomly assigning 200 families to a “control” group that will be monitored but denied access to the HomeBase program for a period of two years.

The idea is immediately disturbing, evoking images of families consigned to homelessness, only to be heartlessly tracked by researchers.  A New York Daily News article about the issue poignantly describes one single mother who “went to HomeBase in August while facing eviction because she owes $3,400 in back rent.” Because she was assigned to the control group, she did not get assistance from HomeBase.

The control group-based design of the study comes from the practice of randomized control trials in medicine. In the medical context, randomized control trials are considered ethical only if there is general doubt about whether the treatment really helps.

If it becomes clear that the treatment really helps, the study must be discontinued. In the language of program evaluators, the HomeBase program is the “treatment.” So is there general doubt about whether HomeBase works?

The HomeBase program’s website lists the program’s services as:

  • Family or tenant/landlord mediation
  • Household budgeting
  • Emergency rental assistance
  • Job training and placement
  • Benefits advocacy (child care, food stamps, tax credits, public health insurance)

I’m willing to doubt that HomeBase’s mediation, budgeting, job training, and advocacy services have an effect on preventing families from falling into homelessness. Mediation may not prevent eviction, budgeting and job training may come too late, and benefits advocacy might be too little to stop a family from becoming homeless. But there is no doubt that emergency rental assistance prevents homelessness, at least temporarily, if it stops a family from being evicted.

New York City’s Mayor’s Management Report suggests that the temporary respite offered by HomeBase may have a permanent effect: 94.6% of the families receiving preventative services in fiscal year 2010 did not enter the shelter system.  This is not a rigorous evaluation by any means: we don’t know how many of those families would have entered the shelter system if they didn’t have help from HomeBase.

In the language of randomized control trials, there is still some doubt as to whether the treatment works.  But it certainly raises the question: do we need an evaluation? Are we willing to risk the lives of 200 families to evaluate a service that, at worst, temporarily prevents homelessness?

I say no. Instead, let’s concentrate our rigorous evaluation efforts on the parts of HomeBase and other programs where our doubt is greater. In evaluating HomeBase, try different combinations of services, pairing rental assistance with different combinations of tenant/landlord mediation, budgeting, job training, and benefits advocacy to see what works best to prevent homelessness.

Experiments in the social sector should focus on what we don’t know and how learning more can help and should avoid putting vulnerable people at risk for little gain.

Photo credit: ranma_tim