As the social sector struggles to measure its impact and make the case that real progress is being made, the LA Unified School District (LAUSD) might have found the easiest and most fool proof way of increasing graduation rates; lower graduation standards. The LAUSD is facing a dropout crisis, and like many social sector organizations, whether government or non-profit, is feeling the pressure to improve outcomes based on a set of measurable indicators. For schools, a fairly important indicator is graduating students.

Yet graduation is actually a proxy for being educated. While econometric models tend to find positive returns to education (more education more money) the diploma itself does not beget a wage increase. Instead, diplomas are a signal that a worker can perform at a certain level.

Although lowering graduation requirements, in this case by potentially allowing students to graduate with fewer credits and lower grade point averages, will increase graduation rates, will it actually make students more educated? And what about those students who graduate under current LAUSD standards, might this proposal cause them harm by allowing students to take fewer classes or earn worse grades?

I’m no education expert, but in my work at Idealistics I work with a range of organizations facing considerable pressure to move the needle on one metric or another. But this approach the LAUSD is considering could set a dangerous precedent for the social sector.

If an employment agency wants to inflate employment statistics, it might conflate temp work with full time jobs. And if a homeless services organization wanted to show an improvement in housing placements, it might refuse to work with chronically homeless persons and only serve those whose homeless-spell would last a month regardless of interventions.

Is someone who is 101% of the poverty line not poor while someone who is 99% of the poverty line poor? Any anti-poverty organization worth its salt would want to help both of these people, regardless of an arbitrary line set by the federal government.

We use thresholds to help describe what we mean by the objectives we aim to address. Lowering our standards and adjusting how we define measurement of our objectives is an easy way to inflate our outcomes, but it is not terribly satisfying and is in no way meaningful.

The LAUSD would be wise to focus its efforts on helping students achieve its vision of an educated youth population rather than lowering its standards to award diluted diplomas. The people we serve are more than binary variables that fail or succeed. Our outcomes metrics are only valuable if they help identifying where we truly succeed and fail. If instead we simply want high scores, then make up any number you want and call it a day.

Photo credit: Craig Morey

While most in philanthropic circles received the report with guarded optimism, the report had decidedly bad news for organizations in the human services field.

From 2009 to 2010, giving to human service organizations actually decreased 5.6%. So while those concerned with philanthropy might have seen some silver lining, or perhaps down right good news in the numbers, from where I sit the findings were all bad.

You see, I’m not in the charitable sector, I’m in the ending poverty business. And while demand for social services have logically increased during a prolonged, hopefully not structural loss of American economic prowess, charitable giving to poverty focused organizations has dipped in tandem.

Economic Losses and Efficiency Gains

The funding environment is not going to improve any time soon, and if our economy does suffer a structural shift, it might not ever. Human services has never been the sexy cause of affluent philanthropists, who tend to favor health, arts, and higher education. Indeed, charitable giving in human services is largely driven by middle and lower-class households.

Rounding out human services is government funding, a lot of it. Of course, that funding is being cut back substantially as governments slash budgets to stave off default.

The stark funding reality in the face of higher demand leaves only one choice. And no, that choice doesn’t have anything to do with $3 dollar Twitter challenges and a hollow promise of social media fundraising success.

We need to get real, and getting real means doubling down on achieving social impact. The only way to do more with less is to get more efficient. Efficiency, of course, cannot be discerned without reliable ways of measuring if what we do actually works.

Housing First, Because it Works

We write a lot about Housing First on this site as a strategy for reducing homelessness. There’s a good reason for that, Housing First actually works, and we have metrics to show it.

Housing First caught on as a solution because it demonstrably houses hurting people while simultaneously decreasing the social cost of homelessness.

This is the kind of innovation we need to pursue, one that is driven by facts and inherently measurable. Sense we can measure the effectiveness of Housing First, we can regularly monitor whether the strategy actually makes social and economic sense. Discerning both the social and fiscal impact of any program must be a prerequisite, especially as funding gets increasingly more difficult to come by.

Ultimately, the changing economic landscape necessitates a shift in the way the social sector operates. I’m afraid the over-emphasis on online fundraising and corporate media challenges only serves to distract from the stark truth; it’s time we hunker down on identifying what works, and cutting what doesn’t.

Of course, if we are to focus our efforts only on solutions that work, we’ll need better ways of measuring success.

Measurement though, I’ve come to learn, has less to do with a solid set of equations and more to do with an organization’s culture. Historically, our sector has a culture of emphasizing increasing revenues over monitoring impact. In the current funding environment, that strategy no longer cuts it.

If our ultimate goal is to end poverty, we have to face the fact that growing the funding pot is not presently a strategy we can pursue. Instead, we will have to do more with less. In order to do that, we have to focus on what works, seriously evaluate our own efforts, and have the courage of our convictions to put the needs of suffering people above the monetary ambitions of our interventions of choice.

Photo credit: Phillipa Willitts

Opponents of quantification argue that the work of the social sector is unique and that social value is not necessarily measurable.

The shortest distance between two points is a straight line, which is why I have long believed that if we are in the business of reducing poverty, then changes in poverty status is the most important (although not the only) social impact metric. However, a new study suggests that poverty reduction might best be measured by the number of people saved from poverty related deaths.

Poverty Related Deaths

The study, published in the American Journal of Health, was a meta study of several reports that tried to determine the number of people whose deaths were attributable to poverty.

Of course, establishing just what poverty related death means is a complicated, if not impossible undertaking. Indeed, the lead author of the study conceded that  “Any time you try to say that death is attributable to a single cause, there’s a problem — all deaths are attributable to many causes. But what we did is just as valid as what was done to establish smoking as a cause of death.”

Considering the now wide spread acceptance that smoking does affect mortality, if the methods used by this study are analogues to the evidence used to link smoking to death, then the researchers might actually be on to something.

Using data from the year 2000, the study indicated that 133,000 people died from poverty related deaths. Compare that to 156,000 deaths from lung cancer in the same year and you start to see where the researchers are headed.

Death is the Common Metric

The social sector gets flack for not having a common unit of measurement. In the business world profit serves as the equalizer across organizations. But if we can measure poverty in terms of death, what can possibly be more equalizing than that? Not all people or organizations achieve profit, but all of us will die.

By framing poverty in terms of death, and if the estimate of poverty related death ranks up there with more celebrated causes like lung cancer, then perhaps we should consider reframing the work that we do.

In this framework, changes in poverty level become intermediate indicators instead of ultimate outcomes.  Instead, the ultimate outcomes metric is the number of lives saved (or lost to) poverty.

But as much as I find the idea intriguing, I’m skeptical of the approach.  I wonder what this exercise really teaches us. Anyone who has been poor or worked directly with people struggling with poverty knows first-hand that being poor increases risk of death.

Furthermore, even if death does make sense as final outcomes metric, it is not useful as an intermittent indicator for planning programs for once someone is dead; there isn’t much you can do for them.

So while this study might have revealed an interesting factoid and possible new way to frame poverty alleviation work to donors, it does not offer anything of value to help improve the impact of poverty interventions. Instead, we’ll have to stick with more tried and true indicators that reveal the trajectory an individual or family is on, rather than their final destination.

Photo credit: funky_abstract

The Department of Homeless Services is planning a study to evaluate the effectiveness of its Homebase program. Homebase helps at-risk individuals and families receive coordinated services designed to prevent homelessness and promote housing stability.

In a study involving 400 families, 200 families will receive Homebase services and 200 families will be denied access to the program for two years in an experiment that has been called “cold-hearted and callous” as well as “cruel and heartless” by City officials.

In an October post for Poverty Insights, I examined the evolution of research ethics in the United States, the protection of human study participants in this country, and suggested issues for New York City officials to consider as they investigate the proposed evaluation of the Homebase program.

The General Welare Committee should also consider two additional events that have taken place since that time.

First, on November 5th, the United States underwent a review of its human rights policies in Geneva by the United Nations Human Rights Council.  One of the main issues identified by the Council was the apparent failure of the U.S. to ensure the human right to housing. Raquel Rolnik, the UN’s top expert on housing rights, toured seven US cities before delivering her critical verbal report to the US State Department.  New York City was one of the cities she visited.

Second, on November 24th, President Barack Obama ordered a comprehensive review of federal and international standards relating to the protection of human study participants in experimental investigations backed by the U. S. pursuant to revelations that prisoners and psychiatric patients in Guatemala were deliberately infected with syphilis from 1946 to 1948 in an apparent effort to test the effectiveness of penicillin.  Questions about the ethical implications of research involving human beings and animals are clearly in the public eye right now.

In light of the statements of the UN and the Obama administration as well as prior ethical concerns, the proposed New York City Homebase evaluation certainly warrants a very thorough investigation by the General Welfare Committee of the New York City Council.

Photo credit: Jake Bouma

These two responsibilities, to provide services and evaluate our work, have come into conflict in the City of New York’s Department of Homeless Services (DHS). DHS is under fire for conducting what is known as a randomized control trial with a sampling of the families they serve.

DHS researchers split service seekers into two groups of 200 families. The first group was provided DHS service as normal, but the second group was simply given a list of non-profits that provide similar services. This second group, referred to as a “control group”, was then told they could not receive services from DHS for the duration of the two-year study.

The experiment was designed to identify what, if any, value was added by DHS’s Homebase program. The Homebase program offers assistance with

• Family or tenant/landlord mediation household budgeting
• Emergency rental assistance
• Job training and placement
• Benefits advocacy (child care, food stamps, tax credits, public health insurance)

According to the DHS website, the department served 8,629 families last fiscal year. Therefore, the control group for this study likely amounts to about 2% of the annual caseload.

Ethical Concerns

There are understandable ethical concerns when conducting research involving vulnerable populations. Speaking to CNN, New York Department of Homeless Services Commissioner Seth Diamond said the clients who participated in the study “had an option; they knew there was a possibility [of being denied services for purposes of the study] and signed a document reflecting that”

While the participants may have signed a consent form, many question the extent to which the consent was truly voluntary.

These are families who came to DHS because they were in desperate situations. Presumably a decision to not sign the consent form was a decision to not receive help from DHS anyway, meaning there was no real choice, rather coerced consent.

This is a serious and important question for our sector, and is not necessarily as black and white as our gut reaction might suggest. Certainly the idea of denying services to someone in need seems wrong, even cruel.

But the hypothesis DHS lays out is that the non-profit referral network might be able to provide the same outcomes as DHS’s Homebase program. And what is to say that people who receive assistance from DHS don’t actually have worse outcomes than people who don’t seek help from the city?

The study seems cruel because we assume that the Homebase program works. Indeed, if the city knew Homebase was effective and denied people help anyway, that would be undoubtedly be cruel. But they don’t.

A Better Way

That said, I am not defending the actions of the New York Department of Homeless Services. My colleague at Idealistics pointed out that when evaluating social service programs, control groups are usually formed by tracking people who did not receive a particular service because the demand exceeded the supply.

What DHS did here is very different.

DHS had the means to provide their service to all who demanded it, but chose not to for the sake of their study. While this style of research is typical in the medical world, where one group of patients is administered an experimental drug then compared to another group that did not receive treatment, it’s not clear that such an approach is appropriate in social sector evaluation.

This is an important issue for the sector to consider. Where do we draw the line in evaluation?

I generally think we do not do enough to evaluate the effectiveness of our programs. Personally, I am surprised to see a situation where we have likely gone too far to evaluate the effectiveness of an intervention.

Was what DHS did unethical? Or do you think conducting a study that impacts a smaller number of families is worth it if it helps us help more people better?

Is there a better way to collect good data, but with less risk to vulnerable populations?

I think so. At the very least, I hope so.

What do you think?

Photo credit: dzingeek

I applaud URM’s decision to try something new, not because I think it will necessarily succeed in improving client outcomes, but because it is an experiment.

Experiments drive progress in the social sector just as they do in any other field, helping us to learn how to achieve better outcomes for our clients.  However, in order for experiments to drive progress, we must accurately measure results to see if they reject or confirm our predictions.

URM CEO Andy Bales, in his recent post on this site, gave three compelling reason why charging shelter residents may achieve better outcomes.

1. Shelter residents who pay may feel more empowered and motivated than their non-paying counterparts, which would drive the paying residents to move on to other kinds of housing more quickly.
2. Paying shelter residents also have an economic incentive to try to speed up the process of their recovery: more time at URM means less money in their pockets.
3. People who are willing to pay for URM’s services might think they will get the most benefit from URM.  The fee helps to screen out people who are not committed to URM’s programs.

All three explanations seem like intuitive reasons that charging for services should improve client outcomes, but they may all be wrong.

Evaluation

The Center for Global Development reviewed multiple studies about the effects of user fees on health and education programs in the developing world. The studies found that people who were willing to pay for services were not necessarily those who needed them most. Furthermore, there was no evidence that paying clients put in more effort than those who received services for free.

Intuitive explanations are often wrong, which is why we need to measure outcomes carefully in our experiments.  When we look at the outcomes of clients who pay for shelter at URM, we need to make sure that we can say, to the best of our ability, “what would have happened to these clients if they hadn’t paid for lodging?”

The best way to find out is to borrow an experimentation technique from the physical sciences: the randomized control trial.  Use random chance to determine who of the eligible clients will be asked to pay and who will not.

Randomized Experiments

Random assignment means there is no characteristic that separates those who pay from those who don’t. Enforcing randomization means we can use the outcomes of the non-paying clients to determine the impact of switching to a pay-to-stay system.

Without randomizing, there is no suitable comparison group.  If, for example, URM used client incomes to determine who should pay, it may be that difference in client incomes, rather than status as paying or non-paying resident, leads to different outcomes.

URM should be praised for experimenting.  Innovation can drive our sector toward better solutions for those most hard-off in society. But in order to know which innovation is effective, we have to measure our experiments carefully.

Photo credit: University of Missouri System