Use the links below to skip to the specific response you wish to view:
California Air Resources Board
February 3, 2021
Elaine M. Howle, CPA
California State Auditor
621 Capitol Mall Suite 1200
Sacramento, California 95814
[submitted via electronics link as directed]
Dear Ms. Howle,
Thank you for the opportunity to review the draft Report 2020-114 California Air Resources Board: Improved Program Measurement Would Help California Work More Strategically to Meet Its Climate Change Goals. The California Air Resources Board (CARB) appreciates the time that your staff has spent with us over the last year to develop the recommendations.
CARB has already started implementing a number of steps to address these recommendations and will be taking future steps as described in the attachment consistent with direction from the Legislature. In the case of some recommendations, it will take time to work through the public process to determine the most efficient and effective mechanisms to collect and analyze the additional data and information that you are recommending, and we are committed to doing so. I also want to note that implementing a number of the recommendations will likely come with an assessment that additional staffing and resources will be needed to fulfill CARB’s ability to deliver them.
You will find attached a more detailed response on how CARB will address the recommendations included in the report.
CARB looks forward to working with your team to track our progress implementing these recommendations and to share that information with the public and the Legislature. If you have any questions, please contact me at (916) 322 7077.
Sincerely,
Richard W. Corey, Executive Officer
Attachment
Attachment
California Air Resources Board (CARB) Responses to California State Auditor Recommendations in Draft Report
California Air Resources Board: Improved Program Measurement Would Help California Work More Strategically to Meet Its Climate Change Goals
Recommendations from Chapter 1
RECOMMENDATION: To improve its ability to isolate each of its incentive programs’ additional GHG reductions, by February 2022 CARB should establish a process to formally identify its incentive programs’ overlap with other programs that share the same objectives. As part of that process, CARB should document how it will attempt to account for the overlap to allow the most accurate program measurement possible.
CARB RESPONSE: CARB will undertake a process to better document the interaction between incentive and regulatory programs and how to refine methods to better account for emission reductions from incentives going forward. CARB will evaluate seeking funding to commission a study to determine where refinements may be needed to our quantification methodologies.
We would like to clarify that, for purposes of tracking progress in meeting health protective federally mandated clean air laws, tools such as EMFAC (which CARB uses to assess emissions levels from specific mobile sources) account for the complementary nature of policies that may impact those sources and avoid overestimating emissions benefits.
RECOMMENDATION: As part of its work to measure both incentive and regulatory programs’ additional GHG reductions, by February 2022 CARB should begin collecting and analyzing the data it needs to assess the extent to which the requirements in its regulatory programs are being exceeded by manufacturers. To the extent applicable, that analysis should focus on the components of the requirements that overlap with CARB’s incentive programs, such as the extent to which manufacturers comply with regulations for heavy-duty vehicles via low- and zero-emission vehicles.
CARB RESPONSE: CARB will include the extent to which regulated parties are exceeding regulatory requirements as we consider how to implement the preceding recommendation.
RECOMMENDATION: To improve its ability to identify the effectiveness of each of its incentive programs in reducing GHG emissions, by August 2021 CARB should develop a process to define, collect, and evaluate data on the behavioral changes that result from each of its incentive programs. Having done so, by February 2022 CARB should collect and analyze relevant survey information for all consumer-focused incentive programs, as well as information about the behavioral effects of programs that other entities offer, such as the federal tax credit.
CARB RESPONSE: CARB has work underway that will help address this recommendation. CARB will also seek public input regarding additional options for conducting this work as a part of the FY 2021-22 Funding Plan for its Low Carbon Transportation incentives including seeking public input around which incentive programs are most suited for additional analysis. CARB currently has work underway that will help with this effort through surveys of participants in projects such as the Clean Vehicle Rebate Project (CVRP), Clean Cars 4 All, Car Sharing, Financing Assistance, and Hybrid and Zero-Emission Truck and Bus Voucher Incentive Project (HVIP) among others.
CARB has also contracted with UC Berkeley to develop an evaluation model/process for CARB to use as a new standard for assessing the effectiveness, sustainability and outcomes of CARB’s clean mobility equity pilot projects for disadvantaged communities and low-income communities. Researchers will identify both community-preferred and research-preferred metrics and evaluation methodologies that can be consistently applied across CARB’s clean transportation equity projects. The study will also result in policy recommendations on successful project elements to inform future transportation equity funding. Results of the UC Berkeley study are expected starting in May 2022.
RECOMMENDATION: To better assist the State in achieving its GHG goals, CARB should use the information we describe above to refine its GHG emissions estimates for its incentive programs in its annual reports to the Legislature, the funding plans approved by its board, and any longer-term planning documents or reports.
CARB RESPONSE: As new data become available through CARB’s implementation of the recommendations in this report, we will use those data to update our GHG quantification methodologies. CARB routinely evaluates and updates the quantification methodologies used to calculate the GHG emission reductions from its incentive programs as new information becomes available, and we will incorporate relevant information obtained through the recommendations in that existing process. CARB uses a public process to update these estimates and publishes the approved California Climate Investments quantification methodologies on its website after a public review period. CARB will use these updated quantification methodologies as it prepares future annual Funding Plans, annual reports to the Legislature on California Climate Investments, and other longer-term planning documents.
RECOMMENDATION: To promote transparency and inform stakeholders, beginning in December 2021, CARB should prepare an annual report for its board and the Legislature on its progress in isolating the GHG emissions reductions attributable to each of its regulatory and incentive programs. As a part of this report, CARB should identify any measurement challenges that persist and highlight any administrative barriers that prevent it from obtaining the information it needs to perform better analysis.
CARB RESPONSE: CARB will report annually on its progress in identifying GHG emission reductions from related transportation incentive and regulatory programs.
RECOMMENDATION: To strengthen the accuracy and integrity of its emissions reduction reporting, CARB should immediately begin retaining all supporting documentation it uses to perform calculations of GHG reductions for its cap-and-trade-funded incentive programs for a period of at least five years. In conjunction with this change, CARB should also document the justification for any instances in which the underlying data it uses to compile its annual reports vary from the information it publishes in those reports.
CARB RESPONSE: CARB is updating its policies and procedures for the various incentives programs it administers to secure additional data related to data reporting and associated records retention.
RECOMMENDATION: To better ensure the accuracy of its program data, by August 2021, CARB should develop a formal schedule and procedures for reviewing the supporting documentation maintained by its program administrators. These procedures, which CARB should begin using with the 2022 annual report, should specify a minimum number of records to review in relation to the program’s size, should specify how staff will collect and maintain evidence to support conclusions, and should be standardized across all of CARB’s incentive programs.
CARB RESPONSE: CARB currently conducts program and desk reviews of its program administrators. In response to this recommendation, CARB will evaluate the need for additional resources and funding to increase the frequency of its reviews and pursue opportunities to standardize that approach.
RECOMMENDATION: To ensure that the State is positioned to assess the status of its sustainable communities program, by April 2021 CARB should report to the Legislature whether it will have a usable source for measuring regional GHG emissions in time for the 2022 report. If CARB believes it may not, it should identify any administrative or bureaucratic barriers it faces in accessing data it needs for the estimates and request relevant action by the Legislature to make those data available.
CARB RESPONSE: The auditor correctly points out that CARB was unable to find a data source to accurately report GHG emissions reductions or vehicle miles travelled (VMT) by region to track SB 375 implementation. Since the 2018 progress report, CARB has initiated a number of efforts to better measure and track SB 375 program progress at a regional level. CARB will provide a status report of this work, including any issues we identify, by April 2021.
Recommendations from Chapter 2
RECOMMENDATION: To ensure that it communicates clearly to the Legislature about the extent to which programs benefit low-income households as the Legislature intended, by March 2022 CARB should begin reporting its spending in low-income communities at the household level wherever possible in its annual report to the Legislature.
CARB RESPONSE: CARB currently implements three incentive projects where participation is limited by household income – Clean Cars 4 All, CVRP, and Financing Assistance for Lower-Income Consumers. CARB collects household income information from participants in each of these projects. As of the most recent California Climate Investments reporting cycle, CARB now reports low income benefits into the California Climate Investments Reporting and Tracking System (CCIRTS) at the household level for all three of these projects. In response to this recommendation, CARB will continue to report low-income benefits at the household level for all future reporting cycles for these projects and for any new projects where consumer participation is limited by household income.
RECOMMENDATION: To better define incentive programs’ impacts beyond GHG emissions reductions, by August 2021 CARB should review its incentive programs to ensure that it has clearly designated which programs focus primarily on socioeconomic benefits. As a result of this process, by February 2022 it should ensure that it includes the benefits expected for each program in its funding plan or other public documents, such as its annual report and individual grant agreements.
CARB RESPONSE: In response to this recommendation, CARB will clarify which Low Carbon Transportation incentive programs provide socioeconomic benefits, including but not limited to public health benefits, green economic opportunities, and greater access to zero emission mobility. CARB’s Low Carbon Transportation equity projects authorized under Health and Safety Code Section 44258.4 (4)(A) are the projects which primarily focus on providing socioeconomic benefits.
CARB considers funding additional projects in each Funding Plan and through a public process, and will consider socioeconomic factors in the development of those projects consistent with direction from the Legislature in its budget appropriations.
RECOMMENDATION: To better demonstrate the socioeconomic benefits that its incentive programs achieve, by February 2022 CARB should do the following:
- Identify clear and measurable metrics it will use to assess each of the socioeconomic benefits it intends its programs to achieve.
- Develop a process to collect data, or use existing data, to measure and report on each metric.
- In its funding plans and annual reports, CARB should report to the Legislature and its board on the metrics.
CARB RESPONSE: In response to this recommendation CARB will work through the public process and with current project grantees, to identify additional socioeconomic metrics associated with clean transportation equity. CARB’s clean transportation equity projects currently incorporate surveys, focus groups, vehicle telematics, and other means of documenting overall project effectiveness, the results of which are used to adaptively manage the projects, address users’ needs, and increase community participation, while also informing future project planning. CARB will evaluate the need to modify future grant solicitations to accommodate relevant additional metrics. The ongoing CARB contracted research by UC Berkeley will provide input to inform this effort. Implementation of this recommendation may include an evaluation of the need for additional resources.
RECOMMENDATION: To provide transparency to the Legislature and other stakeholders, beginning in 2022, using the metrics and data described above, CARB should make funding and design recommendations in its funding plans and annual reports based on which programs are effective in producing socioeconomic benefits and at what cost.
CARB RESPONSE: In response to this recommendation and in accordance with the activities taken under the preceding recommendation, CARB will present its initial findings in the FY 2022-23 Low Carbon Transportation Funding Plan and will continue to report in future annual Funding Plans and annual reports.
RECOMMENDATION: To ensure that the State has reliable information about the extent to which cap-and-trade-funded programs create and support jobs, by August 2021 CARB should begin collecting data on the jobs produced by each of its incentive programs. Where needed, CARB should pursue amendments to its agreements with its program administrators to make reporting this information mandatory. CARB should include an analysis of these jobs data in its annual reports to the Legislature beginning in 2022.
CARB RESPONSE: CARB will work with its grantees for ongoing Low Carbon Transportation projects to collect and report on the direct jobs for each grantee resulting from CARB California Climate Investments funding to the maximum extend feasible. We will work with grantees to revise existing grant agreements where feasible to ensure refined reporting across programs. CARB currently reports on these jobs benefits for the FARMER Program and Community Air Protection Program in the annual California Climate Investments Report.
RECOMMENDATION: To ensure that its incentive programs promote effective and equitable job training, by August 2021 CARB should develop a process to assess which programs should include a job training element. For those programs it identifies, by February 2022 CARB should direct its staff or its external program administrators to collect and report on the quality of job trainings and outcomes experienced by participants, including who received training, the credentials participants received as a result, any actual or expected wages they received as a result of participating in the training or for developing the relevant expertise, and the number of participants from disadvantaged communities or low-income communities and households.
CARB RESPONSE: In response to this recommendation, CARB will seek stakeholder input as part of the annual Low Carbon Transportation Funding Plan process, to determine which programs should include a job training element. Currently, job training and workforce training elements have been included in some clean transportation equity projects and heavy duty demonstration and pilot projects. For projects identified through the annual Funding Plan process as being appropriate to include a job training element, CARB will ensure that grant amendments include the appropriate reporting provisions as noted in the recommendation.
Recommendation from Other Areas Reviewed
RECOMMENDATION: To ensure that it can account for the total costs of its transportation programs, beginning with fiscal year 2021-22 CARB should develop and implement processes to track the administrative costs it incurs to operate each of its transportation programs. After doing so, it should begin including those costs as part of the cost-effectiveness measurements in its annual reports to the Legislature.
CARB RESPONSE: CARB will develop and implement processes to track the administrative costs it incurs to operate each of its transportation programs within FI$Cal and track those costs as part of the cost-effectiveness measurements in its annual reports to the Legislature.
Comments
CALIFORNIA STATE AUDITOR’S COMMENTS ON THE RESPONSE FROM the California Air Resources Board
To provide clarity and perspective, we are commenting on CARB’s response to our audit. The numbers below correspond to the numbers we have placed in the margin of CARB’s response.
We do not understand the rationale for CARB’s statement that it will evaluate seeking funding to commission a study to determine where refinements may be needed to its quantification methodologies. As we discuss throughout Chapter 1, we identified deficiencies with CARB’s methodologies—such as not accounting for overlap between its incentive and regulatory programs—that lead it to overstate those programs’ GHG emissions reductions. Therefore, CARB’s proposed study seems unnecessary given the deficiencies we already identified, and could delay CARB’s implementation of our recommendation.
CARB’s response conflates statewide emissions reporting and its Emission Factor (EMFAC) tool with its measurement of the GHG reductions achieved by its individual transportation programs. During our audit, CARB confirmed that the statewide emissions reductions in CARB’s reporting cannot be attributed to specific programs. The statewide reporting is designed to measure total GHG emissions, but it is not able to assign responsibility for those reductions to individual programs. Further, CARB cannot use the tool to identify or account for overlap in the GHG reductions it projects or reports for each of its transportation programs. Therefore, we stand by our recommendation our recommendation.
Although CARB states that it currently has work underway through surveys to help address this recommendation, our review found that its surveys generally did not address the crucial question of whether participants would have purchased their vehicles without receiving an incentive. Specifically, we determined and show in Table 1, for five programs we reviewed in which CARB provides an incentive payment or other financial assistance to consumers who purchase a low‑ or zero‑emission vehicle—Clean Cars 4 All, CVRP, Financing Assistance, FARMER, and HVIP—CARB only collects survey information about behavior changes for CVRP. Further, we explain that of the four follow‑up surveys CARB has collected for its car‑sharing pilot program, only two contained questions that CARB could use to validate its GHG emissions reduction assumptions about consumers replacing trips in conventional cars. Therefore, as we recommend, CARB should collect and analyze relevant survey information for all consumer‑focused incentive programs to improve its ability to identify the effectiveness of each of its incentive programs in reducing GHG emissions.
We acknowledge CARB’s contract with UC Berkeley for program evaluation on pages 29 and 46 of the audit report. However, the contract with UC Berkeley includes an evaluation of a small subset of CARB’s programs and notably does not include an evaluation of either HVIP or CVRP—two of CARB’s largest incentive programs; it will therefore exclude relevant analysis of the majority of CARB’s programs. Moreover, CARB’s response indicates that results of the UC Berkeley study are expected starting in May 2022, putting those results several months after the completion date of February 2022, which we believe is reasonable to implement our recommendation. Given how crucial participant behavior is to improving CARB’s emissions reduction methodologies, we believe that CARB should take the necessary actions to address our recommendation fully and timely.
As we state above in comment 1, we identified deficiencies with CARB’s quantification methodologies—such as not accounting for overlap between its incentive and regulatory programs—that lead it to overstate those programs’ GHG emissions reductions. Therefore, as we conclude on page 20, the processes CARB describes in its response have not done enough to demonstrate the amount of GHG reductions it projects and measures for its incentive programs. As such, our recommendations reflect our belief that CARB should be proactive in collecting and evaluating the information it needs to better measure programs’ emissions benefits—including information we already identified and named.
CARB’s statement that it currently conducts program and desk reviews of its program administrators overstates its efforts in this area. As we note, CARB has completed desk reviews for only two of the five programs we reviewed that make incentive payments to consumers—HVIP and CVRP. Further, as we also point out, CARB has not conducted any desk reviews for HVIP since 2014, and has conducted only two reviews for CVRP—in 2014 and 2020. Finally, we also conclude that the documentation CARB maintained from the reviews was not sufficient for us to independently determine whether any issues the reviews identified would affect emissions reporting for the programs. The shortcomings of CARB’s current approach are the basis for our recommendation that CARB develop a formal schedule for these reviews, specify how staff will collect and maintain evidence to support conclusions, and standardize the review process across all of CARB’s incentive programs.
To fully implement our recommendation, CARB will need to broaden its planned implementation. Our recommendation that CARB should begin reporting its spending in low‑income communities at the household level wherever possible in its annual report, and not solely for the three incentive programs where participation is limited by household income as CARB’s response states. As we point out, the Legislature has acknowledged that some of the best GHG reduction strategies are those that benefit low‑income households directly, regardless of where those households are located. Therefore, our recommendation is relevant to any CARB program that provides incentive payments at the household or individual level.
Although CARB may have already begun collecting low‑income household data for the Clean Cars 4 All, CVRP, and Financing Assistance programs, this data will not become public until March 2021, when it provides its next annual report to the Legislature. CARB uses the CCIRTS database it mentions in its response to collect data on individual programs for its annual reports. CCIRTS is not itself a report. As such, the activity CARB describes in its response when saying it now reports this information into CCIRTS does not satisfy the recommendation we make on page 53 to publicly report on this information. Therefore, at the point CARB publishes its next annual report, we will evaluate the report to assess CARB’s implementation of this recommendation.
Although most of the incentive programs we reviewed, including CVRP and HVIP, are currently funded under the Low Carbon Transportation portion of the cap‑and‑trade program, our recommendations related to program information and benefits are not limited only to these programs. Rather, our recommendations apply to all of CARB’s transportation programs that target GHG emissions reductions and that may have socioeconomic benefits, or to which cap‑and‑trade reporting guidelines apply.
CARB’s characterization of its efforts to evaluate the socioeconomic benefits of its programs is disingenuous. In fact, as we discuss, CARB does not consistently collect data to determine whether its equity programs actually provide the socioeconomic benefits CARB intends. In an example we discuss in the audit report, the metrics CARB identified for its Financing Assistance program do not allow it to measure the benefits it intends the program to provide. Further, despite repeatedly citing participants’ loan repayment rates as a measure of the Financing Assistance program’s success in its annual funding plans, CARB does not collect data about or report on these rates. As we explain, CARB also does not consistently use the data that are available to demonstrate benefits from programs and determine whether the programs are achieving their goals. Finally, managers for both the Financing Assistance and Clean Cars 4 All programs confirmed that CARB has not made any changes to the programs based on the data provided by program administrators in quarterly reports to CARB. Therefore, we do not agree with CARB’s assertion that it incorporates these types of data to manage the programs and inform future planning.
CARB’s statements are not fully responsive to our recommendation, which states that CARB should begin collecting data on the jobs produced by each of its incentive programs. We explain that CARB’s own 2015 guidelines state that it should track and report the number of job recipients and trainees funded by its programs. As a result, CARB should already be collecting and reporting on this information. Further, as we point out, CARB could seek to amend its grant agreements or, failing that, use its own tool to determine jobs benefits rather than relying on its program administrators to do so. Therefore, CARB should pursue any or all of these reasonable steps to fulfill its own reporting requirements. Finally, although CARB asserts that it reports jobs benefits for the FARMER and Community Air Protection programs, we determined that in its most recent annual report to the Legislature in March 2020, CARB had not reported any job creation numbers for individual programs. Instead, CARB reported aggregate job numbers across all cap‑and‑trade funded programs across multiple agencies. Further, those aggregate numbers were incomplete because, of the nine programs we reviewed for which jobs reporting requirements apply, CARB had collected jobs data in its reporting database for only three at the time of our review.