A weatherization program evaluation story
How do you get 51 state offices and 904 local weatherization agencies to complete detailed surveys during the largest production push the weatherization community has ever seen?
The Energy Center tackled that question through our role in the national evaluation of the Weatherization Assistance Program. In 2009 the passage of the American Recovery and Reinvestment Act funneled $5 billion into the Weatherization Assistance Program, increasing production from roughly 100,000 homes a year to more than 600,000 homes over a three-year period. ARRA also funded a national evaluation to measure energy savings, cost effectiveness and non-energy benefits produced by the program.
At the same time that state and local agencies were gearing up to meet aggressive new production targets, our evaluation team was fielding a series of time-intensive surveys and other requests to collect the information needed to measure program performance. With a focus on relationship-building, patience, persistence and some flexibility, we managed to achieve a 100 percent response rate to data requests fielded at the state level and response rates above 90 percent for requests made of local weatherization agencies.
The key to the success of this truly massive data collection effort was building relationships at the state and local levels. Oak Ridge National Laboratory laid the foundation by soliciting input on the evaluation plan from a formal Network Committee of weatherization thought leaders from around the country. Before fielding any data requests, the Energy Center established relationships with two national organizations that play an important role in the weatherization community, NASCSP and NCAF.
The Energy Center hired five case managers to serve as the front line for state and agency outreach and communication efforts. Each case manager was assigned a portfolio of states and served as a designated point of contact for the state weatherization office and all local agencies in each state. The case manager team fielded data requests, answered questions from respondents, and gently but persistently encouraged survey completion in the timeframe necessary to meet evaluation needs.
Data collection for the first stage of the evaluation wrapped up in 2011, with 51 state offices and 878 weatherization agencies completing 2800 surveys. These surveys contained more than 3600 unique variables and 9.2 million data points ranging from high-level information about program administration at the state and local levels to detailed records on the treatment of individual weatherized homes.
The national evaluation of the Weatherization Assistance Program is led by ORNL, with project management oversight by APPRISE, a Princeton-based nonprofit research firm. Since its inception in 1976, the Weatherization Assistance Program has supported the weatherization of more than 6.4 million low income households.
The Energy Center is coordinating the collection of data from state offices and local weatherization agencies, and leading technical studies that measure weatherization impacts on use of delivered fuels (fuel oil and propane), indoor air quality, air conditioning electricity use, as well as the persistence of energy savings from weatherization measures over time.
The Energy Center will repeat these data collection activities in 2012 as the second phase of the evaluation will measure program performance under ARRA. Although the ARRA-period production push of 2009 - 2011 is winding down, the weatherization network faces new challenges in the significant cutbacks in funding that have impacted state offices and local weatherization agencies. The relationships developed during the first phase will be key to collecting the data necessary to ensure robust results on program accomplishments during the ARRA period.
Evaluation reports will be published on the ORNL website.