Report 2020-114 Recommendation 3 Responses

Report 2020-114: California Air Resources Board: Improved Program Measurement Would Help California Work More Strategically to Meet Its Climate Change Goals (Release Date: February 2021)

Recommendation #3 To: Air Resources Board, State

To improve its ability to identify the effectiveness of each of its incentive programs in reducing GHG emissions, by August 2021 CARB should develop a process to define, collect, and evaluate data on the behavioral changes that result from each of its incentive programs. Having done so, by February 2022 CARB should collect and analyze relevant survey information for all consumer-focused incentive programs, as well as information about the behavioral effects of programs that other entities offer, such as the federal tax credit.

Annual Follow-Up Agency Response From October 2023

CARB continues to implement changes in response to this recommendation. CARB staff developed a draft process to define, collect, and evaluate data related to behavioral changes associated with projects funded in the FARMER, Low Carbon Transportation, and AB 617 Community Air Protection Programs, as required by August 2021.

For the consumer-based projects, staff developed a draft streamlined survey for the vehicle purchase incentive projects, including the Clean Vehicle Rebate Project, Financing Assistance Program, and Clean Cars 4 All, and another survey was developed for CARB's mobility projects. CARB is coordinating with internal and external researchers for feedback and guidance on these surveys and coordinating across projects to ensure data can be compared, where appropriate. Because of delays in accessing appropriate technology to administer the surveys, CARB staff aims to launch the surveys by the end of 2023 and begin collecting and analyzing data starting in 2024, to inform project development going forward. CARB staff also developed an improved survey for Clean Cars 4 All which provides a performance evaluation, as directed by Section 44125.5(c) of AB 630 (Statutes of 2017, Chapter 636), while simultaneously providing data to inform future goal-setting.

In the FARMER Program, CARB included direction in the fiscal year (FY) 2021-22 grants with the administering air districts to survey funding recipients on the behavioral impacts of these incentives in order to evaluate program effectiveness. CARB has begun receiving responses as part of air districts' semi-annual reports and will continue to evaluate responses on these and others as they are received.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From October 2022

CARB continues to implement changes in response to this recommendation. CARB staff developed a draft process to define, collect, and evaluate data related to behavioral changes associated with projects funded in the FARMER, Low Carbon Transportation, and AB 617 Community Air Protection Programs, as required by August 2021.

For the consumer-based projects, staff developed a draft streamlined survey for the vehicle purchase incentive projects, including the Clean Vehicle Rebate Project, Financing Assistance Program, and Clean Cars 4 All, and another survey was developed for CARB's suite of mobility projects. CARB is coordinating with internal and external researchers for feedback and guidance on these surveys and coordinating across projects to ensure data can be compared, where appropriate. CARB staff also developed an improved survey for Clean Cars 4 All to address direction in AB 630 (Statutes of 2017). Because of delays in accessing appropriate technology to administer the streamlined survey, CARB staff is working with administrators and University of California partners to find an efficient way to launch, collect, and evaluate data collected from the survey. CARB aims to begin collecting and analyzing relevant survey information in 2023, to help inform project development going forward.

In the FARMER Program, CARB included direction in the FY 2021-22 grants with the administering air districts to survey funding recipients on the behavioral impacts of these incentives in order to evaluate program effectiveness. CARB expects to start receiving these survey data before the end of 2022 as part of air districts' semi-annual reports to CARB.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

1-Year Agency Response

CARB staff developed a draft process to define, collect, and evaluate data related to behavioral changes associated with projects funded in the FARMER, Low Carbon Transportation, and AB 617 Community Air Protection Programs, as required by August 2021.

In the FARMER Program, CARB is including in the FY 2021-22 funding cycle grants with the administering air districts a requirement to survey funding recipients on the behavioral impacts of these incentives in order to evaluate program effectiveness. CARB expects to start receiving these survey data later in 2022 as part of air districts' semi-annual reports to CARB once air districts start spending their FY 2021-22 FARMER funds.

For the consumer-based projects, staff continues to develop a streamlined survey that can be administered and managed by CARB. One survey is currently in development for the vehicle purchase incentive projects, including the Clean Vehicle Rebate Project, Financing Assistance Program, and Clean Cars 4 All, and another survey is being developed for CARB's suite of mobility projects. CARB is coordinating with internal and external researchers for feedback and guidance on these surveys, and coordinating across projects to ensure data can be compared, where appropriate. Staff is also in the process of acquiring appropriate technology to deploy the survey. Staff aims to begin collecting and analyzing relevant survey information by mid-2022, to help inform project development going forward.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

CARB provided documentation supporting its response that it has developed or is requiring survey tools for some of its transportation programs. To fully implement the recommendation, CARB must implement relevant tools for all incentive programs, collect and analyze resulting data along with relevant metrics related to other entities' programs, and make determinations about the programs' effectiveness as a result of its analysis.

6-Month Agency Response

CARB staff has developed a draft process to define, collect, and evaluate data related to behavioral changes associated with projects funded by Low Carbon Transportation, FARMER, and AB 617 Community Air Protection Program, as required by August 2021. Staff continues to work to identify the best approaches for measuring and evaluating data on behavioral changes that result from these programs and determine the best way to report such information.

For the consumer-based projects, staff is in the process of developing streamlined surveys that can be administered and managed by CARB. One survey is currently in development for the vehicle purchase incentive projects, including the Clean Vehicle Rebate Project, Financing Assistance Program, and Clean Cars 4 All, and another survey is being developed for CARB's suite of mobility projects. CARB is coordinating with internal and external researchers for feedback and guidance on these surveys, and coordinating across projects to ensure data can be compared, where appropriate. Staff intends to have surveys developed and begin administration of them by Fall 2021, and collect and begin analyzing relevant survey information by February 2022. Staff intends to use information from evaluations conducted to help inform development of projects beginning in Fiscal Year 2022-23.

California State Auditor's Assessment of 6-Month Status: Pending

CARB provided a document outlining its intent to collect and measure relevant data for the programs we identified in our audit report. However, as it indicates in its response, CARB is still in the process of identifying specific approaches to measuring and evaluating relevant data..

60-Day Agency Response

CARB staff are reviewing existing methods for evaluating the behavioral impacts of projects funded by Low Carbon Transportation, FARMER and AB 617 Community Air Protection Program. Staff are working to identify the best approaches for measuring and evaluating data on behavioral changes that result from these programs and determine the best way to report such information.

For the consumer-based projects, staff are in the process of developing streamlined surveys that can be administered and managed by CARB. One survey is currently in development for the vehicle purchase incentive projects, including the Clean Vehicle Rebate Project, Financing Assistance Program, and Clean Cars 4 All, and another survey is being developed for CARB's suite of mobility projects. CARB is coordinating with internal and external researchers for feedback and guidance on these surveys, and coordinating across projects to ensure data can be compared, where appropriate. Staff intends to have surveys developed and begin administration of them by August 2021.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2020-114

Agency responses received are posted verbatim.