Report 2020-114 All Recommendation Responses

Report 2020-114: California Air Resources Board: Improved Program Measurement Would Help California Work More Strategically to Meet Its Climate Change Goals (Release Date: February 2021)

Recommendation #1 To: Air Resources Board, State

To improve its ability to isolate each of its incentive programs' additional GHG reductions, by February 2022 CARB should establish a process to formally identify its incentive programs' overlap with other programs that share the same objectives. As part of that process, CARB should document how it will account for the overlap to allow the most accurate program measurement possible.

Annual Follow-Up Agency Response From October 2023

For over a decade, CARB has used multiple policy approaches, including incentives and regulations, to reduce greenhouse gas (GHG) emissions and transition to zero-emission vehicles. To respond to CSA's recommendation to separate GHG reductions of individual strategies, CARB executed a contract with researchers at the University of California specializing in environmental economics and policy. During the contract period, CARB met with the researchers biweekly to provide necessary program data for their investigation. Findings published in June 2023 concluded that both regulations and incentives led to significant and direct GHG reductions. However, participant survey data was insufficient to distinguish isolated incentive program impacts. Moreover, the report highlighted how incentive programs provided "second order" benefits beyond the direct benefits of additional zero-emission vehicle sales. For instance, incentives increased consumer demand, which spurred automaker investment, and by 2022 enabled CARB to set ambitious zero-emission sales targets that aligned with the Governor's Executive Order N-79-20 to phase-out combustion vehicle sales by 2035. Consequently, the report highlighted, "because the various ZEV programs work in concert, the role of any one policy is impossible to disentangle with precision from among various consumer and business decisions to purchase or lease ZEVs." CARB is thus unable to further refine its methodologies to isolate GHG reductions from its complementary programs. Nevertheless, if methods to approximate the individual contributions of incentive programs become available, moving forward, CARB will explore opportunities to further refine GHG quantification methodologies.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Although CARB contracted with UC Berkeley to attempt to identify the impact of overlapping regulatory and incentive programs, CARB acknowledges that it was unable to distinguish between incentive and regulatory impacts on emissions. Further, the report produced by UC Berkeley indicates that the research team is currently reviewing available data on ZEV purchases to better estimate the number of ZEVs that are purchased or leased as a direct result of regulatory and incentive programs.

The report notes that the incentive programs overlap with regulations, but does not indicate how CARB will account for the overlap to allow the most accurate program measurement possible and isolate the reductions attributable to each program.

We acknowledge that CARB has taken efforts to be responsive to the recommendation, but in the absence of a formal process for identifying and accounting for overlap, CARB has not addressed all aspects of the recommendation.

Annual Follow-Up Agency Response From October 2022

Meeting California's ambitious climate targets requires a mix of policy approaches, including regulatory mandates, incentives, consumer education, and infrastructure investment. Together, these complementary policies interact and influence consumer behavior and the vehicle market to achieve GHG emission reductions greater than single policies would achieve alone. In addition, the combination of policies that achieve GHG reductions also advance environmental justice and multiple environmental and social goals. Therefore, it is challenging to attribute emissions reductions to individual programs. Addressing the recommendation to establish a formal process will require additional data and analysis to estimate the impacts of individual components of CARB's integrated policy approach.

As mentioned in the One Year Update, to obtain the additional data and analyses that are needed to address CSA's recommendation, CARB executed a contract with a team of researchers at the University of California who specialize in technological innovation systems theory, behavioral research, and environmental policy. The contract was executed in December 2021 and will take approximately two years to complete. Since January 2022, CARB has been meeting with the researchers on a biweekly basis as they review California's portfolio policy approach related to transportation and assess the effectiveness of incentives in changing consumer behavior, achieving further GHG emissions reductions, and advancing social equity. CARB expects to receive an interim report in December 2022. In 2023, the researchers will develop quantification methods for existing consumer-based incentive programs and provide recommendations for long-term data collection methods and analysis tools to quantify the emission benefits of individual incentive programs.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

1-Year Agency Response

Meeting California's ambitious climate targets requires a mix of policy approaches, including mandates, incentives, consumer education, and infrastructure investment. These policies induce further GHG emission reductions, advance environmental justice, and concurrently address multiple environmental and social goals. However, their interrelated operation and the complexity of consumer behaviors make it challenging to attribute emissions reductions to individual programs. Addressing this challenge requires additional data and analysis to separate the impacts of CARB's existing incentives on consumer behavior from regulations and other programs.

To address CSA's recommendation, CARB executed a contract with a team of researchers at the University of California who specialize in technological innovation systems theory, behavioral research, and environmental policy. The objective of this contract is to better define GHG emission benefits of individual CARB incentive programs. The researchers are reviewing California's portfolio policy approach related to transportation and assess the effectiveness of incentives in changing consumer behavior, achieving further GHG emissions reductions, and advancing social equity. The researchers will also develop quantification methods for existing consumer-based incentive programs and provide recommendations for long-term data collection methods and analysis tools to quantify the emission benefits of incentive programs. The contract was executed in December 2021 and will take approximately 2 years to complete. CARB began meeting with the researchers biweekly in January 2022 to discuss CARB's programs and provide regulatory and incentive program data necessary for the researchers' analysis. CARB expects to receive interim results in Summer 2022.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

In addition to the the contract CARB references in its response, CARB provided evidence that it has been meeting with the researchers to provide them the information necessary to conduct their work. Therefore, although the substantive portions of this recommendation are still outstanding, CARB has demonstrated partial implementation.

6-Month Agency Response

Meeting California's ambitious climate targets requires a mix of policy approaches, including mandates, incentives, consumer education, and infrastructure investment. The complementary nature of these policies has the benefit of inducing further GHG emission reductions, advancing environmental justice in California's underserved communities, and concurrently addressing multiple environmental and social goals. However, this complementary nature and the complexity of consumer behaviors make it very challenging to attribute emissions reductions to each individual program. Addressing this challenge requires additional data collection and analysis on the impacts of CARB's existing incentives on consumer behavior aside from those resulted from regulations and other programs.

CARB is developing a contract with university researchers to review California's portfolio policy approach related to transportation and its effectiveness. The contract is intended to inform development of long-term data collection methods and analysis tools that can assess the effectiveness of incentives in changing consumers' behaviors and disentangle the emission benefits of incentives and other programs. Results from this contract will be used to develop methods that quantify the benefits of CARB's light- and heavy-duty consumer-focused ZEV incentive programs. The contract will likely take approximately 2 years to complete, and CARB expects to receive interim results and report the researchers' findings to CSA starting in Summer 2022.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

Meeting California's ambitious climate targets requires a mix of policy approaches, including technology forcing measures like the ZEV mandate, incentives, consumer education, and infrastructure investment. The complementary nature of these policies has the benefit of inducing further GHG emission reductions that concurrently address multiple environmental and social goals, including providing benefits to the overall economy. To meet Recommendation #1, we will conduct further investigation through further data collection and analysis. We are committed to establishing a process to refine how we analyze the effect of each policy.

As part of this implementation, CARB plans to formally define objectives for each of its incentive programs and identify quantifiable metrics to assess progress towards these objectives. CARB plans to designate or hire additional staff with expertise in economics and data science to further study and evaluate potential overlaps among various incentive and regulatory programs. CARB also intends to commission an extramural contract to collect necessary information needed to attribute greenhouse gas emission and socio-economic benefits to each individual program. CARB staff will utilize data collected through this contract to inform the process and methodologies implemented by the agency moving forward. CARB anticipates needing additional resources to further enhance this process.

California State Auditor's Assessment of 60-Day Status: Pending

Fully implementing this recommendation may require CARB to collect data that is relevant to its program design that it does not currently collect. However, the primary purpose of the recommendation is to ensure that CARB recognizes, and accounts for, overlap among the key objectives of its incentive and regulatory programs. Our work identified many of these key objectives for the programs we reviewed, including where existing overlap is clear. Therefore, CARB should first focus on coordinating among its programs and divisions to identify and account for the objectives its programs are already pursuing. Our review did not identify any reason this process would require additional resources beyond those CARB already has in place to manage the design and implementation of its programs. If CARB believes such resources are necessary, it must clearly demonstrate why the process of implementing this recommendation requires personnel or expertise it does not currently have available with its existing resources.

Recommendation #2 To: Air Resources Board, State

As part of its work to measure both incentive and regulatory programs' additional GHG reductions, by February 2022 CARB should begin collecting and analyzing the data it needs to assess the extent to which the requirements in its regulatory programs are being exceeded by manufacturers. To the extent applicable, that analysis should focus on the components of the requirements that overlap with CARB's incentive programs, such as the extent to which manufacturers are complying with regulations for heavy-duty vehicles via low- and zero-emission vehicles.

1-Year Agency Response

For CARB's light-duty ZEV Regulation, CARB staff began disclosing additional public data starting with the 2020 reporting year to help identify manufacturer over compliance. These data can be found in CARB's 2020 ZEV credits annual disclosure report (https://ww2.arb.ca.gov/sites/default/files/2021-12/2020_zev_credit_annual_disclosure_ac.pdf), which was released in December 2021. Disclosures for this reporting year and subsequent years include a more comprehensive overview of manufacturer compliance. Updates include: the regulatory credit requirement by manufacturer, the number of eligible vehicle sales and associated credits generated by make and model, a complete accounting of each manufacturer's credit generation and balance from the prior reporting year to the current year, and the industry-aggregated requirement and credits earned from 2014 through the current reporting year. This public disclosure and related data will allow stakeholders (including contracted researchers) to better understand how manufacturers are complying with the ZEV Regulation in terms of vehicles and regulatory credits relative to the regulation requirements for each year.

For CARB's heavy-duty GHG regulations, CARB staff has begun analyzing data to assess over-compliance with the California Phase 1 GHG regulation for 2019 and 2020 model year heavy-duty vehicles. Findings include, but are not limited to, the number of vehicles produced in California, and the number of certified manufacturers that generated conventional and advanced technology credits due to producing heavy-duty battery electric, fuel cell, and hybrid-electric vehicles. CARB will continue to do this annually. This information will be used to inform CARB's work in isolating benefits of our incentive programs.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

CARB provided documentation of the extent to which manufacturers are overcomplying with required standards for the vehicles affected by CARB's consumer incentive programs. CARB's analysis demonstrates that manufacturers of light- and heavy-duty vehicles are using low- and zero-emissions vehicles to exceed the regulatory requirements. CARB's continued analysis of this behavior by manufacturers will help it better identify the effects of its incentive programs on the sale of these vehicles.

6-Month Agency Response

To identify the extent to which manufacturers are over complying with CARB's light-duty ZEV Regulation, CARB staff will include additional data that original equipment manufacturers (OEMs) disclose to CARB for purposes of complying with the regulation starting with the 2020 reporting year. This additional data will be posted in CARB's ZEV Credits Public Disclosure Document (https://ww2.arb.ca.gov/our-work/programs/advanced-clean-cars-program/zev-program/zero-emission-vehicle-credit-balances), which is typically released annually in the fall of each year. Additional disclosures for each reporting year will include, but are not limited to: the regulatory credit requirement by manufacturer, the number of eligible vehicle sales by make and model, and the associated credits generated by make and model for each manufacturer. This data will allow stakeholders to better understand how manufacturers are complying with the ZEV Regulation in terms of vehicles and regulatory credits relative to the regulation requirements for each reporting year.

For CARB's heavy-duty GHG regulations, CARB staff has begun analyzing data to assess over compliance with Phase 1 GHG regulation for 2019 model year heavy-duty vehicles. Staff will continue to analyze data for 2020 model year heavy-duty vehicles. Findings including, but not limited to, the number of vehicles produced in California, and the number of certified manufacturers that generated conventional and advanced technology credits due to producing heavy-duty battery electric, fuel cell, and hybrid-electric vehicles would be provided once available. CARB will continue to do this annually in future years.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

CARB provided documentation indicating it has begun analyzing the extent to which heavy-duty vehicle manufacturers are using low- and zero-emission technology to comply with emissions regulations. According to CARB's documentation, certain manufacturers are using low-and zero-emissions technologies to over comply with the regulatory requirement. As we state in our report, such overcompliance could indicate the additional effect of incentive programs focusing on the same vehicles. CARB should use this information, and the information it plans to collect for its ZEV regulations, to inform its work on implementing our related recommendations for isolating the benefits of its incentive programs.

60-Day Agency Response

CARB is committed to assessing the extent to which manufacturers are exceeding requirements in our regulatory programs. CARB staff is currently investigating the most appropriate methodology to assess the extent to which the requirements in the ZEV Regulation are being exceeded by manufacturers in number of vehicles. Beginning in the 2020 reporting year, CARB staff will annually analyze over compliance with the light-duty ZEV Regulation. Findings will be provided in the fall of each year to CARB's incentives team for use in complying with Recommendation 1. CARB staff is also evaluating what additional data and resources are necessary to complete the recommended assessment for our heavy-duty programs as they are implemented.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #3 To: Air Resources Board, State

To improve its ability to identify the effectiveness of each of its incentive programs in reducing GHG emissions, by August 2021 CARB should develop a process to define, collect, and evaluate data on the behavioral changes that result from each of its incentive programs. Having done so, by February 2022 CARB should collect and analyze relevant survey information for all consumer-focused incentive programs, as well as information about the behavioral effects of programs that other entities offer, such as the federal tax credit.

Annual Follow-Up Agency Response From October 2023

CARB continues to implement changes in response to this recommendation. CARB staff developed a draft process to define, collect, and evaluate data related to behavioral changes associated with projects funded in the FARMER, Low Carbon Transportation, and AB 617 Community Air Protection Programs, as required by August 2021.

For the consumer-based projects, staff developed a draft streamlined survey for the vehicle purchase incentive projects, including the Clean Vehicle Rebate Project, Financing Assistance Program, and Clean Cars 4 All, and another survey was developed for CARB's mobility projects. CARB is coordinating with internal and external researchers for feedback and guidance on these surveys and coordinating across projects to ensure data can be compared, where appropriate. Because of delays in accessing appropriate technology to administer the surveys, CARB staff aims to launch the surveys by the end of 2023 and begin collecting and analyzing data starting in 2024, to inform project development going forward. CARB staff also developed an improved survey for Clean Cars 4 All which provides a performance evaluation, as directed by Section 44125.5(c) of AB 630 (Statutes of 2017, Chapter 636), while simultaneously providing data to inform future goal-setting.

In the FARMER Program, CARB included direction in the fiscal year (FY) 2021-22 grants with the administering air districts to survey funding recipients on the behavioral impacts of these incentives in order to evaluate program effectiveness. CARB has begun receiving responses as part of air districts' semi-annual reports and will continue to evaluate responses on these and others as they are received.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From October 2022

CARB continues to implement changes in response to this recommendation. CARB staff developed a draft process to define, collect, and evaluate data related to behavioral changes associated with projects funded in the FARMER, Low Carbon Transportation, and AB 617 Community Air Protection Programs, as required by August 2021.

For the consumer-based projects, staff developed a draft streamlined survey for the vehicle purchase incentive projects, including the Clean Vehicle Rebate Project, Financing Assistance Program, and Clean Cars 4 All, and another survey was developed for CARB's suite of mobility projects. CARB is coordinating with internal and external researchers for feedback and guidance on these surveys and coordinating across projects to ensure data can be compared, where appropriate. CARB staff also developed an improved survey for Clean Cars 4 All to address direction in AB 630 (Statutes of 2017). Because of delays in accessing appropriate technology to administer the streamlined survey, CARB staff is working with administrators and University of California partners to find an efficient way to launch, collect, and evaluate data collected from the survey. CARB aims to begin collecting and analyzing relevant survey information in 2023, to help inform project development going forward.

In the FARMER Program, CARB included direction in the FY 2021-22 grants with the administering air districts to survey funding recipients on the behavioral impacts of these incentives in order to evaluate program effectiveness. CARB expects to start receiving these survey data before the end of 2022 as part of air districts' semi-annual reports to CARB.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

1-Year Agency Response

CARB staff developed a draft process to define, collect, and evaluate data related to behavioral changes associated with projects funded in the FARMER, Low Carbon Transportation, and AB 617 Community Air Protection Programs, as required by August 2021.

In the FARMER Program, CARB is including in the FY 2021-22 funding cycle grants with the administering air districts a requirement to survey funding recipients on the behavioral impacts of these incentives in order to evaluate program effectiveness. CARB expects to start receiving these survey data later in 2022 as part of air districts' semi-annual reports to CARB once air districts start spending their FY 2021-22 FARMER funds.

For the consumer-based projects, staff continues to develop a streamlined survey that can be administered and managed by CARB. One survey is currently in development for the vehicle purchase incentive projects, including the Clean Vehicle Rebate Project, Financing Assistance Program, and Clean Cars 4 All, and another survey is being developed for CARB's suite of mobility projects. CARB is coordinating with internal and external researchers for feedback and guidance on these surveys, and coordinating across projects to ensure data can be compared, where appropriate. Staff is also in the process of acquiring appropriate technology to deploy the survey. Staff aims to begin collecting and analyzing relevant survey information by mid-2022, to help inform project development going forward.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

CARB provided documentation supporting its response that it has developed or is requiring survey tools for some of its transportation programs. To fully implement the recommendation, CARB must implement relevant tools for all incentive programs, collect and analyze resulting data along with relevant metrics related to other entities' programs, and make determinations about the programs' effectiveness as a result of its analysis.

6-Month Agency Response

CARB staff has developed a draft process to define, collect, and evaluate data related to behavioral changes associated with projects funded by Low Carbon Transportation, FARMER, and AB 617 Community Air Protection Program, as required by August 2021. Staff continues to work to identify the best approaches for measuring and evaluating data on behavioral changes that result from these programs and determine the best way to report such information.

For the consumer-based projects, staff is in the process of developing streamlined surveys that can be administered and managed by CARB. One survey is currently in development for the vehicle purchase incentive projects, including the Clean Vehicle Rebate Project, Financing Assistance Program, and Clean Cars 4 All, and another survey is being developed for CARB's suite of mobility projects. CARB is coordinating with internal and external researchers for feedback and guidance on these surveys, and coordinating across projects to ensure data can be compared, where appropriate. Staff intends to have surveys developed and begin administration of them by Fall 2021, and collect and begin analyzing relevant survey information by February 2022. Staff intends to use information from evaluations conducted to help inform development of projects beginning in Fiscal Year 2022-23.

California State Auditor's Assessment of 6-Month Status: Pending

CARB provided a document outlining its intent to collect and measure relevant data for the programs we identified in our audit report. However, as it indicates in its response, CARB is still in the process of identifying specific approaches to measuring and evaluating relevant data..

60-Day Agency Response

CARB staff are reviewing existing methods for evaluating the behavioral impacts of projects funded by Low Carbon Transportation, FARMER and AB 617 Community Air Protection Program. Staff are working to identify the best approaches for measuring and evaluating data on behavioral changes that result from these programs and determine the best way to report such information.

For the consumer-based projects, staff are in the process of developing streamlined surveys that can be administered and managed by CARB. One survey is currently in development for the vehicle purchase incentive projects, including the Clean Vehicle Rebate Project, Financing Assistance Program, and Clean Cars 4 All, and another survey is being developed for CARB's suite of mobility projects. CARB is coordinating with internal and external researchers for feedback and guidance on these surveys, and coordinating across projects to ensure data can be compared, where appropriate. Staff intends to have surveys developed and begin administration of them by August 2021.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #4 To: Air Resources Board, State

To better assist the State in achieving its GHG goals, CARB should use the information we describe to refine its GHG emissions estimates for its incentive programs in its annual reports to the Legislature, the funding plans approved by its board, and any longer-term planning documents or reports.

Annual Follow-Up Agency Response From October 2023

CARB continues to implement changes in response to this recommendation. Work to address this recommendation is contingent on the outcome of CARB's efforts to address recommendations 1 through 3, including the research contract described in CARB's response to recommendation 1. That work is currently under contract and the associated task is expected to be completed in late Fall of 2023. The outcomes of these analyses will determine how CARB will address this recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2022

CARB continues to implement changes in response to this recommendation. Work to address this recommendation is contingent on the outcome of CARB's efforts to address recommendations 1 through 3, including the research contract described in CARB's response to recommendation 1. That work is currently under contract. The outcomes of these analyses will determine how CARB will address this recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

1-Year Agency Response

Work to address this recommendation is contingent on the outcome of CARB's efforts to address recommendations 1 through 3, including the research contract described in CARB's response to recommendation 1. That work is currently under contract, and interim results are expected starting in Summer 2022. The outcomes of these analyses will determine how CARB will address this recommendation.

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

CARB is committed to refining our emission estimate methodology as more data and information become available. As is noted in the recommendation, the data collected and analyzed through implementing the previous 3 recommendations will be used to refine the GHG emissions estimates for our incentive programs for this recommendation. There is not yet new information available upon which to update GHG emissions estimates from CARB's incentives programs.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

CARB is committed to refining our emission estimate methodology as more data and information becomes available. As is noted in the recommendation, the data collected and analyzed through implementing the previous 3 recommendations will be used to refine the GHG emissions estimates for our incentive programs for this recommendation.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #5 To: Air Resources Board, State

To promote transparency and inform stakeholders, beginning in December 2021, CARB should prepare an annual report for its board and the Legislature on its progress in isolating the GHG emissions reductions attributable to each of its regulatory and incentive programs. As a part of this report, CARB should identify any measurement challenges that persist and highlight any administrative barriers that prevent it from obtaining the information it needs to perform better analysis.

1-Year Agency Response

CARB is committed to maintaining a transparent process for informing stakeholders on progress made toward isolating the emission benefits of regulatory and incentive programs. To report progress made in 2021, CARB prepared a memo for the CARB Board and a letter for the Joint Legislative Audit Committee and submitted them on December 17, 2021. CARB will continue reporting on an annual basis its progress in quantifying the benefits of CARB programs to the CARB Board and the Legislature.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

The letters to the Joint Legislative Audit Committee and its board that CARB references in its response include information about the contracting activities described in CARB's response to Recommendation #1. As this recommendation states, these reports should continue annually and, when CARB's review has produced results, include information about those results as well as any measurement or administrative challenges that persist.

6-Month Agency Response

CARB is committed to maintaining a transparent process for informing stakeholders on progress made toward disentangling the emission benefits of regulatory and incentive programs. To report the progress, CARB plans to prepare a memo for the Board and submit a letter to the Joint Legislative Audit Committee in December 2021. Following the recommendation, the memo and letter will identify any measurement challenges that persist and highlight any administrative barriers.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

CARB is committed to maintaining a transparent process for informing stakeholders on progress made toward isolating GHG emission reductions attributable to each regulatory and incentive program. CARB is in the process of identifying the appropriate mechanism for annually reporting this information and how to overcome administrative barriers, such as necessary resources, needed to implement recommendations 1 through 4.

California State Auditor's Assessment of 60-Day Status: Pending

We understand CARB's response to indicate that it plans to meet the December 2021 date in the recommendation for providing an update to its board and the Legislature. With respect to CARB's reference to necessary resources, implementing recommendations 1 through 4 will require CARB to collect additional data and more effectively analyze data it currently possesses. However, our review did not identify any clear instances of insufficient resources as the cause for the program measurement issues we identified. Therefore, if CARB believes such resources are necessary, it must clearly demonstrate why the process of implementing those recommendations requires personnel or expertise it does not currently have available with its existing resources.

Recommendation #6 To: Air Resources Board, State

To strengthen the accuracy and integrity of its emissions reduction reporting, CARB should immediately begin retaining all supporting documentation it uses to perform calculations of GHG reductions for its cap-and-trade-funded incentive programs for a period of at least five years. In conjunction with this change, CARB should also document the justification for any instances in which the underlying data it uses to compile its annual reports vary from the information it publishes in those reports.

60-Day Agency Response

Policies and procedures that outline the steps taken to prepare and report GHG reductions for Low Carbon Transportation, FARMER, and CAP incentives have been expanded and updated to include the requirement that all supporting documentation, including any documentation necessary to perform GHG calculations, shall be maintained for at least five years. The procedures include a step-by-step guide for reporting emission reductions, as well as instructions for documentation of each step taken at the time of emission reduction reporting. Finally, the procedures address how to document variances between the underlying data used to compile the annual reports and the information published in those reports.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

CARB provided copies of the policies and procedures for our review. These policies and procedures cover cap-and-trade funded programs and fulfill the retention and documentation components of our recommendation.

Recommendation #7 To: Air Resources Board, State

To better ensure the accuracy of its program data, by August 2021 CARB should develop a formal schedule and procedures for reviewing the supporting documentation maintained by its program administrators. These procedures, which CARB should begin using with the 2022 annual report, should specify a minimum number of records to review in relation to the program's size, should specify how staff will collect and maintain evidence to support conclusions, and should be standardized across all of CARB's incentive programs.

6-Month Agency Response

Policies and Procedures for the Low Carbon Transportation, FARMER, and CAP Incentive Programs have been developed to include procedures on how staff will conduct and maintain evidence of program oversight by the program administrator/grantee; which staff refer to as desk reviews. These policies and procedures have been recently updated to include more clearly defined formal review schedules and minimum numbers of records to be reviewed for each program.

Though each incentive program is unique in appropriation amount, administration process, maximum incentive level per project, etc., the procedures for conducting reviews are standardized across all programs where feasible. Additionally, reviews for all applicable funding programs will be combined when a review is conducted on administrators of multiple CARB incentive programs.

CARB will begin using these updated policies and procedures prior to the 2022 annual report.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

CARB's 6-month submission, and the revised procedures it provided to us, address the issues with the frequency and scope of CARB's planned reviews that we identified with its 60-day response.

60-Day Agency Response

Policies and Procedures for the Low Carbon Transportation Program, FARMER, and CAP Incentives have been updated to include procedures on how staff will conduct and maintain evidence of program oversight by the program administrator/grantee; which staff refer to as desk reviews. Though the number of program administrators/grantees and number of records and is dependent on staffing resources and the risk associated with each project administrator, procedures include necessary steps and timelines to ensure appropriate documented oversight going forward.

California State Auditor's Assessment of 60-Day Status: Pending

The procedures CARB provided do not implement key elements of our recommendation. They do not establish a formal schedule for conducting reviews of CARB's various programs. As such, the procedures do not make clear how quickly CARB will actually conduct reviews of all its programs and program administrators. In addition, the procedures do not all specify a minimum number of records to review in relation to the programs' size. Finally, CARB's procedures make the timing and scope of the reviews contingent on available resources. The purpose of our recommendation was for CARB to commit to a clear process for performing that review. Until it does so, the recommendation will not be implemented.

Recommendation #8 To: Air Resources Board, State

To ensure that the State is positioned to assess the status of the Sustainable Communities program, by April 2021 CARB should report to the Legislature whether it will have a usable source for measuring regional GHG emissions in time for the 2022 report. If CARB believes it may not, it should identify any administrative or bureaucratic barriers it faces in accessing data it needs for the estimates and request relevant action by the Legislature to make those data available.

60-Day Agency Response

Since completing the 2018 Progress Report on California's Sustainable Communities and Climate Protection Act, CARB initiated efforts to better measure and track SB 375 progress at a regional level. One of these efforts is tracking additional region-specific metrics in each metropolitan planning organization's (MPO) sustainable community strategy (SCS) starting with third-round SCSs. While these metrics are not direct measures of GHG emissions or VMT, they provide additional context on progress toward expected plan outcomes to meet GHG targets set by CARB.

Additionally, CARB is exploring using retail fuel sales data from the California Energy Commission in conjunction with statewide fuel sales data from the California Department of Tax and Fee Administration to estimate regional VMT and GHG emissions. Early analysis indicates that this could be a useful approach for MPOs whose VMT is relatively contained within the region. Results from this approach will not provide a complete picture for all regions, but it will be an improvement to CARB's previous analysis. This is the approach we will use for the 2022 Progress Report.

For future reports, CARB is exploring the viability of using big data. Big data could provide a complete accounting of VMT and where it is occurring, and an insight on travel behavior and trends through the consolidation of location data from millions of mobile devices. CARB anticipates needing new staff and contract resources to acquire and analyze big data for this purpose.

We notified Assemblymember Rudy Salas, Chair of JLAC, of this work on April 20, 2021.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

CARB's response, and its letter to the Legislature informing it about CARB's planned course of action, do not satisfy our recommendation that CARB inform the Legislature whether CARB will have a usable source for measuring regional GHG emissions in time for the 2022 report. Instead, the letter indicates that CARB is exploring the use of fuel sales data, but cautions that this approach may not provide "a complete picture for all regions".

Because of this ambiguity, we reached out to CARB for clarification. When we did so, CARB confirmed that it will use the fuel sales data in the 2022 report and indicated that it has found additional data sources that it believes will address the limitations described in its initial response. Therefore, because CARB has committed to this approach, we consider the recommendation to be fully implemented.

Recommendation #9 To: Air Resources Board, State

To ensure that it communicates clearly to the Legislature about the extent to which programs benefit low-income households as the Legislature intended, by March 2022 CARB should begin reporting its spending in low-income communities at the household level wherever possible in its annual report to the Legislature.

1-Year Agency Response

As previously reported in the 60-day and 6-month responses, there are three projects where financial incentives are paid directly to individual households: the Clean Vehicle Rebate Project, Clean Cars 4 All, and Financing Assistance for Lower-Income Consumers. These are the projects for which CARB has determined it is possible to report low-income spending at the household level. CARB already collects household income information from participants in each of these projects, and CARB now reports low income benefits at the household level into the California Climate Investments Reporting and Tracking System (CCIRTS) for all three projects. CARB will also report this information for any additional, future projects that provide financial benefits directly to individual households.

To fully address CSA's recommendation, CARB reported the spending that benefits low-income households for each of these projects in the Fiscal Year (FY) 2021-22 Funding Plan for Clean Transportation Incentives, published on October 8, 2021 and approved by CARB's Board on November 19, 2021. For the most recent reporting period, the percent of funding that went to low-income households was 21 percent for CVRP, 98 percent for Clean Cars for All, and 91 percent for financing assistance as shown on pages 50, 64, and 68 of the Funding Plan, respectively (https://ww2.arb.ca.gov/sites/default/files/2021-10/fy21-22_fundingplan.pdf). CARB will continue to report this information each annual Funding Plan moving forward.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

As noted in the 60-day response, there are three projects where financial incentives are paid directly to individual households: the Clean Vehicle Rebate Project, Clean Cars 4 All, and financing assistance. These are the projects for which CARB has determined it is possible to report low-income spending at the household level. CARB already collects household income information from participants in each of these projects, and CARB now reports low income benefits at the household level into the California Climate Investments Reporting and Tracking System (CCIRTS) for all three projects.

To address CSA's recommendation, CARB will start reporting the spending that benefits low-income households (along with low-income communities and disadvantaged communities) for each of these projects in the Fiscal Year (FY) 2021-22 Clean Transportation Funding Plan and in each annual funding plan thereafter. CARB will also report low-income benefits at the household level in the future for any new projects where financial incentives are provided directly to California households.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

CARB has reviewed its suite of GHG incentive programs to determine where financial incentives are provided directly to California households, rather than to fleets, businesses, school districts, public agencies, or to communities. There are three projects where financial incentives are paid directly to individual households: the Clean Vehicle Rebate Project, Clean Cars 4 All, and financing assistance. These are the projects for which CARB has determined it is possible to report low-income spending at the household level. CARB already collects household income information from participants in each of these projects, and CARB now reports low income benefits at the household level into the California Climate Investments Reporting and Tracking System (CCIRTS) for all three projects.

CSA correctly noted to CARB that data reported into CCIRTS is not necessarily reported publicly with the same granularity. In the case of low-income benefits, the California Climate Investments annual report shows the combined spending that benefits low-income communities and low-income households. It does not report each separately. To address CSA's recommendation, CARB will start reporting the spending that benefits low-income households (along with low-income communities and disadvantaged communities) for each of these projects in the FY 2021-22 Clean Transportation Funding Plan and in each annual funding plan thereafter. CARB will also report low-income benefits at the household level in the future for any new projects where financial incentives are provided directly to California households.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #10 To: Air Resources Board, State

To better define incentive programs' impact beyond GHG emissions reductions, by August 2021 CARB should review its incentive programs to ensure that it has clearly designated which programs focus primarily on socioeconomic benefits. As a result of this process, by February 2022 it should ensure that it includes the benefits expected for each program in its funding plan or other public documents, such as its annual report and individual grant agreements.

Annual Follow-Up Agency Response From October 2022

This recommendation has been fully implemented. As previously reported in the 6-month and one-year responses, CARB staff has reviewed its incentive programs and identified six clean transportation equity projects that focus primarily on socioeconomic benefits. These projects are: Clean Cars 4 All, Financing Assistance for Lower-Income Consumers, Clean Mobility Options, Clean Mobility in Schools, the Sustainable Transportation Equity Project, and the Agricultural Workers Vanpool Project (currently inactive). CARB identified and reported the expected project benefits, corresponding metrics and evaluation methods for each of the projects (aside from the inactive Agricultural Workers Vanpool Project) in the FY 2021-22 Funding Plan for Clean Transportation Incentives, published on October 8, 2021 and approved by CARB's Board on November 19, 2021 (https://ww2.arb.ca.gov/sites/default/files/2021-10/fy21-22_fundingplan.pdf).

CARB has re-evaluated and refined project benefits, including to identify socioeconomic benefits, for these project write-ups in the proposed FY 2022-23 Funding Plan for Clean Transportation Incentives going before CARB's board in November 2022. The Funding Plan is available at: https://ww2.arb.ca.gov/sites/default/files/2022-10/proposed_fy2022_23_funding_plan_final.pdf.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

CARB shared with us its revised expected socioeconomic benefits for the programs in question. We agree that this recommendation has now been fully implemented.

1-Year Agency Response

As previously reported in the 6-month response, CARB staff has reviewed its incentive programs and identified six clean transportation equity projects that focus primarily on socioeconomic benefits. These projects are: Clean Cars 4 All, Financing Assistance for Lower-Income Consumers, Clean Mobility Options, Clean Mobility in Schools, the Sustainable Transportation Equity Project, and the Agricultural Workers Vanpool Project (currently inactive). CARB identified and reported the expected project benefits, corresponding metrics and evaluation methods for each of the projects (aside from inactive Agricultural Workers Vanpool Project) in the FY 2021-22 Funding Plan for Clean Transportation Incentives, published on October 8, 2021 and approved by CARB's Board on November 19, 2021 (https://ww2.arb.ca.gov/sites/default/files/2021-10/fy21-22_fundingplan.pdf).

CARB will re-evaluate and refine project benefits for these projects and for any new projects that focus primarily on socioeconomic benefits in each annual Funding Plan moving forward.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

As part of its 6-month response, CARB shared with us the socioeconomic benefits, associated metrics, and planned evaluation process for relevant transportation programs. However, we believe--and CARB agrees--that these benefits and the associated metrics under Recommendation #11 require further refinement in order to be effective measures of the program's effects. Therefore, this recommendation is partially implemented, and we will continue to work with CARB toward full implementation.

6-Month Agency Response

CARB staff has reviewed its incentive programs and identified six clean transportation equity projects that focus primarily on socioeconomic benefits. These projects are: Clean Cars 4 All, Financing Assistance for Lower-Income Consumers, Clean Mobility Options, Clean Mobility in Schools, Sustainable Transportation Equity Project, and the Agricultural Workers Vanpool Project (currently inactive). Staff has identified project benefits, corresponding metrics and evaluation methods for inclusion in the FY 2021-22 Funding Plan for Clean Transportation Investments which will be published in October 2021. Staff will complete an initial analysis of these benefits throughout the remainder of 2021 and include results in future Funding Plans.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

CARB provided documentation that indicates it has begun identifying the expected benefits it expects each program to provide. We will further assess CARB's progress at the time of its one-year response.

60-Day Agency Response

Many of our programs do provide some socioeconomic benefits in addition to the GHG and air quality benefits. Our clean transportation equity projects, in general, are the ones where socioeconomic benefits are the greatest priority. CARB staff is reviewing its incentive programs to identify methods to measure the socioeconomic benefits for programs that focus on providing socioeconomic benefits as cobenefits, and will complete that work by August 2021. In addition, CARB staff is developing a format in which to include the expected benefits for each program in upcoming funding plans.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #11 To: Air Resources Board, State

To better demonstrate the socioeconomic benefits that its incentive programs achieve, by February 2022 CARB should do the following:

- Identify clear and measurable metrics it will use to assess each of the socioeconomic benefits it intends its programs to achieve.
- Develop a process to collect data, or use existing data, to measure and report on each metric.
- In its funding plans and annual reports, CARB should report to the Legislature and its board on the metrics.

Annual Follow-Up Agency Response From October 2023

In the FY 2022-23 Funding Plan for Clean Transportation Incentives, CARB identified metrics for projects mentioned in Recommendation 10. Since then, CARB and its grantees have developed standardized data collection processes including participant surveys, user testimonials, and vehicle telematics. The metrics CARB identified include:

Clean Cars 4 All (CC4A) - Increase vehicle reliability measured by future disruptions of service based on the model year of the retired vehicle and that of the incentivized vehicle, and improve access to employment, goods, and services based on user surveys and testimonials.

Financing Assistance for Lower-Income Consumers - Improve ability of low-income or disadvantaged community members to purchase a clean vehicle using a low-interest loan, measured by the number of clean vehicles purchased by low-income consumers and the number of loans facilitated by the program.

Clean Mobility Options - Improve access to goods, services, and workforce training and development measured through participant surveys and vehicle telematics data.

Clean Mobility in Schools - Improve students' access to schools and transportation cost savings made possible by investments.

Sustainable Transportation Equity Project (STEP) - Improve priority populations' access to key destinations and transportation cost savings made possible by investments.

All active projects are now collecting data. The new Financing Assistance and Statewide CC4A program launches in early 2024 as a needs-based model and new metrics will be added. Staff plan to report on the analysis of these metrics as part of the FY 2024-25 Funding Plan process where robust data is available.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From October 2022

CARB continues to implement changes in response to this recommendation. In the proposed FY 2022-23 Funding Plan for Clean Transportation Incentives, CARB identified clear and measurable metrics for projects mentioned in Recommendation 10. CARB and project grantees are developing standardized data collection processes through participant surveys, user testimonials, and vehicle telematics. Metrics for each project identified in CARB's response to Recommendation 10 include:

Clean Cars 4 All - Increased vehicle reliability measured by future disruptions of service based on the model year of the retired vehicle and that of the incentivized vehicle, and improved access to employment, goods, and services based on user surveys and testimonials.

Financing Assistance for Lower-Income Consumers - Improved ability of low-income or disadvantaged community member to purchase a clean vehicle, using a low-interest loan, without having a vehicle to retire.

Clean Mobility Options - Improved access to goods, services, and workforce training and development opportunities measured through participant surveys and vehicle telematics data.

Clean Mobility in Schools - Improve students' access to schools and transportation cost savings made possible by investments.

Sustainable Transportation Equity Project - Improve priority populations' access to key destinations and transportation cost savings made possible by investments.

In future Funding Plans, starting with the FY 2023-24 Funding Plan, staff will report on the analysis of these metrics for each project where robust data is available.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

We agree that CARB has identified the expected benefits for the programs in question under recommendation #10. We will continue to work with CARB to assess the data it collects from these programs and the conclusions it draws from such data about the programs' effectiveness.

1-Year Agency Response

In the FY 2021-22 Funding Plan for Clean Transportation Incentives published on October 8, 2021 and approved by CARB's Board on November 19, 2021, CARB identified clear and measurable metrics for projects mentioned in Recommendation 10. CARB and project grantees are developing standardized socioeconomic benefit metrics and implementing data collection processes through participant surveys and testimonials and vehicle telematics. Examples of metrics for each project identified in CARB's response to Recommendation 10 include:

Clean Cars 4 All - increased vehicle reliability measured by future disruptions of service based on the model year of the retired vehicle and that of the incentivized vehicle, and improved access to employment and goods and services based on user surveys and testimonials.

Financing Assistance for Lower-Income Consumers - increased number of clean vehicles in disadvantaged communities based on the increased number of cleaner vehicles purchased by disadvantaged community residents utilizing the program.

Clean Mobility Options - improved access to goods and services and workforce training and development opportunities measured through participant surveys and vehicle telematics data.

Clean Mobility in Schools - improved visibility and acceptance of zero-emission mobility options based on school district staff feedback and vehicle telematics.

Sustainable Transportation Equity Project - travel cost savings based on number of subsidies, value of each subsidy, and average fare of each clean transportation service.

These are described further in the FY 2021-22 Funding Plan. In future Funding Plans starting with the FY 2022-23 Funding Plan, staff will report on the analysis of these metrics for each project.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

Please see our assessment of Recommendation #10, which also concerns the metrics discussed in this recommendation.

6-Month Agency Response

CARB staff has identified clear and measurable metrics for each of the projects mentioned in Recommendation 10 and will include those in the FY 2021-22 Funding Plan which will be published in October 2021. The FY 2021-22 Funding Plan will also include the process developed for each project to collect additional data or use existing data to measure and report on each metric that has been identified. In future Funding Plans, staff will report on the analysis of these metrics for each of the projects.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

CARB provided evidence that it has begun identifying metrics by which it plans to measure the socioeconomic benefits its programs are intended to achieve. We will continue to assess CARB's progress in implementing this recommendation when CARB submits its one-year response.

60-Day Agency Response

As mentioned above, CARB staff is currently evaluating incentive programs that provide socioeconomic benefits as cobenefits. CARB staff will review current program data that is collected and will use a public process for the Funding Plan to help identify clear and measurable metrics including how the data will continue to be collected, analyzed, and measured. CARB will include information on these metrics in future funding plans and/or annual reports, as appropriate.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #12 To: Air Resources Board, State

To provide transparency to the Legislature and other stakeholders, beginning in 2022 and using the metrics and data described above, CARB should make funding and design recommendations in its funding plans and annual reports based on which programs are effective in producing socioeconomic benefits and at what cost.

Annual Follow-Up Agency Response From October 2023

CARB staff continues to develop draft valuation methodologies for use in evaluating socioeconomic benefits on a cost basis. Multiple years of data are required for this type of analysis across all projects, and sufficient data will not be available to complete an analysis until the timing of the FY 2025-26 Funding Plan, at the earliest. At that time, CARB will endeavor to make funding and design recommendations based on the socioeconomic benefits and at what cost.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2022

CARB continues to implement changes in response to this recommendation. CARB staff is in the early stages of developing draft valuation methodologies that will be used to evaluate the socioeconomic benefits on a cost basis. CARB will describe the process by which metrics and data will be folded into the funding and design decision making process in the FY 2023 24 Funding Plan. Multiple years of data are required for this type of analysis, and sufficient data likely will not be available to complete an analysis until the FY 2025-26 Funding Plan after which CARB can start making funding and design recommendations based on the socioeconomic benefits and at what cost.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

As noted in our assessment of recommendation #11, we will continue to work with CARB to assess the data it collects from these programs and the conclusions it draws from that data about the programs' effectiveness.

1-Year Agency Response

As noted in CARB's responses to recommendations 10 and 11, CARB has already identified clear and measurable metrics for each of its incentive projects that focus primarily on socioeconomic benefits as well as the process to collect data to measure and report on each metric. These are described in the FY 2021-22 Funding Plan for Clean Transportation Incentives published on October 8, 2021 and approved by CARB's Board on November 19, 2021. CARB staff is developing a draft cost valuation methodology that it will use to start evaluating the socioeconomic benefits on a cost basis. CARB will start making funding and design recommendations based on this analysis starting in the FY 2022-23 Funding Plan. CARB expects this approach to be refined and expanded over multiple funding cycles. For projects in the early stages of implementation where data are not yet available, staff will include in the FY 2022-23 Funding Plan the process by which metrics and data will be folded into the funding and design decision making process when sufficient data do become available.

California State Auditor's Assessment of 1-Year Status: Pending

See our responses to CARB's updates on the status of Recommendations 10 and 11. The issues raised in those responses are relevant to fully implementation this recommendation.

6-Month Agency Response

CARB staff has begun data analysis for some of the projects where existing data are available, and will make funding and design recommendations based on this analysis starting in the FY 2022-23 Funding Plan, including providing information on which programs are effective in producing socioeconomic benefits and at what cost. For projects in the early stages of implementation where data are not yet available, staff will include in the FY 2022-23 Funding Plan the process by which metrics and data will be collected, measured, and reported.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

Once the actions from recommendations 10 and 11 have progressed to the point where better data is available, CARB staff will utilize that data to help inform funding and design recommendations in its funding plans and annual reports.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #13 To: Air Resources Board, State

To ensure that the State has reliable information about the extent to which cap-and-trade-funded programs create and support jobs, by August 2021 CARB should begin collecting data on the jobs produced by each of its incentive programs. Where needed, CARB should pursue amendments to its agreements with its program administrators to make reporting this information mandatory. CARB should include an analysis of these jobs data in its annual reports to the Legislature beginning in 2022.

Annual Follow-Up Agency Response From October 2023

With the publication of the 2022 Annual Report to the Legislature California Climate Investments Report in April 2023, CARB has now publicly reported jobs data for all of its California Climate Investments funded incentive programs. CARB continues to implement changes in response to this recommendation. As noted in previous updates, CARB has already included jobs reporting requirements in the grant agreements for all projects solicited since August 2017 consistent with the requirements in the Funding Guidelines for Agencies Administering California Climate Investments. CARB has also already entered all jobs data provided by grantees into the California Climate Investments Reporting and Tracking System. In addition, CARB already publicly reports the cumulative modeled jobs supported by California Climate Investments for all Low Carbon Transportation investments since fiscal year 2014-15 in its annual Funding Plan for Clean Transportation Incentives and updates these estimates annually. Accordingly, this recommendation is now fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

CARB provided evidence that, with its latest CCI report, it has included jobs estimates for all relevant programs.

Annual Follow-Up Agency Response From October 2022

CARB continues to implement changes in response to this recommendation. CARB has included jobs reporting requirements in the grant agreements for all projects solicited since August 2017 consistent with the requirements in the Funding Guidelines for Agencies Administering California Climate Investments. CARB has already entered the jobs data provided by grantees into the California Climate Investments Reporting and Tracking System (CCIRTS). CARB reported these jobs data in the Annual Report to the Legislature on California Climate Investments (Annual Report) for all projects launched over a year ago. For more recent projects that did not have sufficient data to include in the 2021 Annual Report, CARB will include all reported jobs data in the 2022 Annual Report to be published in March 2023. In addition, CARB already publicly reports the cumulative modeled jobs supported by California Climate Investments for all Low Carbon Transportation investments since fiscal year 2014 15 in its annual Funding Plan for Clean Transportation Incentives. CARB updates these modeled jobs supported estimates annually.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

As we described in our previous assessment of this recommendation, and as CARB confirms here, it has yet to provide comprehensive reporting to the Legislature about the jobs benefits of all relevant programs. Therefore, this recommendation remains partially implemented.

1-Year Agency Response

For the 36 project grant solicitations released since August 2017, 21 already include jobs reporting requirements, 13 are in the process of being amended to include these requirements, and just 2 projects will not include them (one of which is already closed, and the other closes soon). CARB staff is in now in the process of coordinating the collection and reporting of these data and will commence reporting jobs data from these projects by utilizing the California Climate Investments Reporting and Tracking System (CCIRTS), as it has already been doing for the FARMER and Community Air Protection Programs. At the start of 2022, CARB released a memo reminding project grantees of the need to report jobs data for this purpose. CARB plans to publish the Low Carbon Transportation project data as part of the 2022 update of the Annual Report to the Legislature on California Climate Investments.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

CARB's response acknowledges that some of its grant agreements are in the process of being updated to include the job reporting requirements, that its efforts to collect this data are ongoing, and that it has yet to provide comprehensive reporting to the Legislature about the jobs benefits of all relevant programs. Therefore, this recommendation remains partially implemented.

6-Month Agency Response

CARB has already identified where jobs data has not yet been reported for its programs, has amended ongoing grant agreements to incorporate this task where feasible, and is in now in the process of coordinating the collection and reporting of these data. CARB staff will commence reporting jobs data from these projects by utilizing the California Climate Investments Reporting and Tracking System (CCIRTS), as it has already been doing for the FARMER and Community Air Protection Programs. CARB is still evaluating the appropriate mechanism for reporting this information annually.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

CARB provided a list of its grant agreements, which indicates which agreements it has already amended in response to our recommendation. The list also indicates CARB's intent to amend every other such agreement that does not currently contain a requirement for reporting jobs information.

60-Day Agency Response

We have provided refresher training to all project leads on jobs reporting requirements. CARB is also pursuing amendments, where needed, to ensure that program administrators are reporting the necessary jobs information. CARB will also strive to update historical data related to jobs reporting back to August 2017. CARB currently reports on these jobs benefits for the FARMER program and Community Air Protection Program the California Climate Investment Reporting and Tracking System (CCIRTS). CARB is evaluating the appropriate mechanism for reporting this information annually.

California State Auditor's Assessment of 60-Day Status: Pending

Because State law directs CARB, where feasible, to use cap-and-trade funds in a way that fosters job creation in California, our recommendation is that CARB provide jobs data for its programs in its annual report to the legislature about the impacts of the State's cap-and-trade spending. CARB's own funding guidelines for statewide cap-and-trade programs, which support this annual reporting, require reporting on programs' impacts on jobs. Therefore, if CARB decides to report on job information in a different context, it should ensure that report is comprehensive of all its cap-and-trade funded programs and that it provides the report to the Legislature not less than annually.

Recommendation #14 To: Air Resources Board, State

To ensure that its incentive programs promote effective and equitable job training, by August 2021 CARB should develop a process to assess which programs should include a job training element. For those programs it identifies, by February 2022 CARB should direct its staff or its external program administrators to collect and report on the quality of job trainings and outcomes experienced by participants, including who received training, the credentials participants received as a result, any actual or expected wages they received as a result of participating in the training or for developing the relevant expertise, and the number of participants from disadvantaged communities or low-income communities and households.

Annual Follow-Up Agency Response From October 2023

CARB continues to implement changes in response to this recommendation. As part of the annual Low Carbon Transportation Funding Plan development process, and in coordination with the California Energy Commission, CARB staff identified those projects that offered the greatest potential for including job training and workforce development elements. The following projects in the FY 2021-22 and proposed FY 2022-23 Funding Plan for Clean Transportation Incentives were identified to have education, job training, and workforce training and development elements:

- STEP,

- Clean Mobility in Schools,

- Clean Mobility Options,

- Access Clean California,

- The Workforce Training and Development Projects, and

- Heavy-Duty Advanced Technology Demonstration and Pilot Projects.

CARB has started executing projects and anticipates the remaining grants will be executed for these projects in the first quarter of 2024. The grants will include requirements for grantees and program administrators to collect information on the quality of job trainings and outcomes experienced by participants, including the information recommended in the audit:

- Who received training,

- The credentials participants received as a result,

- Expected wages that participants will receive as a result of the training or for developing the relevant expertise, and

- The number of participants from priority populations (i.e., disadvantaged communities and low-income communities and households).

CARB has started receiving reports from grantees and program administrators that have executed grants and expects to begin receiving reports from the remaining grantees and program administrators not yet in grants summarizing this information in 2024.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From October 2022

CARB continues to implement changes in response to this recommendation. The following projects in the Fiscal Year (FY) 2021-22 and proposed FY 2022-23 Funding Plan for Clean Transportation Incentives were identified to have education, job training, and workforce development elements:

Sustainable Transportation Equity Project (STEP),

Clean Mobility in Schools,

Clean Mobility Options,

Access Clean California,

The Workforce Training and Development Projects, and

Heavy-Duty Advanced Technology Demonstration and Pilot Projects.

CARB anticipates grants will be executed for these projects in 2023. The grants will include requirements for grantees and program administrators to collect information on the quality of job trainings and outcomes experienced by participants, including the information recommended in the audit:

Who received training,

The credentials participants received as a result,

Expected wages that participants will receive as a result of the training or for developing the relevant expertise, and

The number of participants from priority populations (i.e., disadvantaged communities and low-income communities and households).

CARB expects to begin receiving reports from grantees and program administrators summarizing this information in 2024.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

As we confirmed during our previous assessment of this recommendation, CARB has identified the programs it expects to provide job training benefits. However, the substantive elements of the recommendation, such as executing the contracts in question, are still outstanding.

1-Year Agency Response

CARB approved the FY 2021-22 Funding Plan for Clean Transportation Incentives on November 19, 2021. Through the Funding Plan development process, CARB identified projects that will include a workforce or jobs training element. The Funding Plan highlights high level details on the types of data to be collected during the life of the project to ensure that job training goals are well defined. Specific details on the data to be collected will be determined as part of the subsequent public process through the building out of competitive solicitations, developing the implementation manual, or updating existing implementation manuals. These requirements will be detailed in grant agreements upon execution.

The FY 2021-22 Funding Plan also includes a Workforce Training and Development project. This project, in addition to incorporating metrics such as jobs, jobs training sessions, networking workshops, as well as capacity building metrics, will also report on increases in workforce training and development in priority populations. It will include requirements for program administrators to identify direct and measurable community benefits, such as socioeconomic, job access, zero-emission technology, environmental literacy, and other quality of life and social impact improvements. This evaluation effort will be done in parallel with CARB's other clean transportation and mobility investments to determine relevant data and lessons that can be applied across programs. This includes soliciting and evaluating ideas on how CARB and its partners should measure and report workforce training and development benefits and providing strong feedback loops for policy and program adjustments based on findings.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

CARB's most recent funding plan identifies the programs it expects to provide job training and related elements as socioeconomic benefits. Although the substantive elements of this recommendation are still outstanding, CARB has therefore demonstrated partial implementation.

6-Month Agency Response

CARB is currently in the process of developing the FY 2021-22 Funding Plan for Clean Transportation Incentives - a process which includes a series of public workshops and work group meetings where stakeholders have the opportunity to provide input on proposed projects. Projects that include a workforce or jobs training element will now and henceforth be identified during the public process and clearly denoted in the annual Funding Plan. The Plan will highlight high level details on the types of data CARB will collect during the projects. Details of the data collected will be determined through a public process while building out the competitive solicitation, developing the implementation manual, or updating existing implementation manuals. Specific data requirements will be included in grant agreements upon execution.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

CARB will identify in the FY 2021-22 Clean Transportation Funding Plan which of the Low Carbon Transportation projects will include a job trainings element. We will continue to identify these projects via this process moving forward. In general, equitable job training is always a component of competitive solicitations that are focused on demonstrations or pilots.

CARB has always included a job training element with respect to heavy duty demonstration and pilot projects. We will continue to include this element as part of our solicitation process and will require that project administrators provide the data parameters identified by the auditors for all future projects.

California State Auditor's Assessment of 60-Day Status: Pending

As we state in our report, when CARB does include job training as a program benefit, it does not always require its program administrators to provide detailed information about that training or how it benefits participants. These gaps in measurement include determining who participates and whether they come from disadvantaged or low-income communities or households. Therefore, CARB's statement that equitable job training is always a component of these programs is inaccurate. In implementing this recommendation, CARB must go beyond its current practices by ensuring that job training goals are well defined and that CARB actually collects and measures data to support its claims about program benefits.

Recommendation #15 To: Air Resources Board, State

To ensure that it can account for the total costs of its transportation programs, beginning with fiscal year 2021-22 CARB should develop and implement processes to track the administrative costs it incurs to operate each of its transportation programs. After doing so, it should begin including those costs as part of the cost-effectiveness measurements in its annual reports to the Legislature.

Annual Follow-Up Agency Response From October 2022

CARB continues to implement changes in response to this recommendation. CARB's Administrative Services Division (ASD) created employee timekeeping codes and FI$Cal chart fields to capture programs and functions, tasks, and projects common across all CARB transportation programs. New expenditures, contracts, purchase order, and labor charges are being recorded to specific programs to provide the total costs for each transportation program. ASD implemented a more detailed timesheet coding to better track associate staff time on CARB's various greenhouse gas emission reduction transportation programs with the underlying fees/revenues that fund those administrative costs and is working to finalize fee codes for new fees in each year's budget. That will help CARB better estimate the costs of it incurs to operate each of its transportation programs. In the final phase, ASD is working to identify remaining programs that should be tracked, and which functional tasks, projects, or specific actions are not part of a program but should still be tracked using FI$Cal chart fields.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

CARB previously provided us its new timekeeping codes and confirmed for us that its development of the more detailed timesheet coding it references in its response is still ongoing.

1-Year Agency Response

To better account for the total costs of its transportation programs, including administrative costs, CARB's Administrative Services Division (ASD) requested employee timekeeping and item coding information from each of CARB's Divisions and established 100+ new timesheet and item codes for use beginning with fiscal year 2021-22. Additionally, all new contracts, purchase orders, labor on timesheets, etc. will record costs specific to each program to provide the total costs of transportation programs. The codes were established in FI$Cal and the Tempo timekeeping program used by CARB on July 30, 2021 and were used beginning with the July 2021 timesheets. ASD runs monthly reports for Divisions with transportation programs so they can see program costs and that staff are charging time appropriately. Additional timekeeping and item codes can be developed if necessary. Divisions will use information from the new coding structure when reporting to the Legislature in the Fall of 2022 for the fiscal year 2023-24 budget cycle.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

CARB's Administrative Services Division (ASD) requested Service Location and Labor Code information from each of CARB's Divisions with transportation programs and established 100+ new codes for their transportation programs for use beginning with FY 2021-22. Going forward, all new contracts, purchase orders, labor on timesheets, etc. will be recorded at the program-level of detail ensuring CARB accounts for the total costs of its transportation programs. The codes were established in FI$Cal and the Tempo timekeeping program used by CARB on July 30, 2021 and are ready for staff to use beginning with their July 2021 timesheets. ASD will run monthly reports to send to Divisions with transportation programs so they can see program costs and that staff are charging time appropriately. Additional Service Locations and Labor Codes can be developed if necessary.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

CARB provided a list of time reporting codes that cover both incentive-based and regulatory transportation programs. To the extent CARB consistently and accurately uses these codes, they will help CARB obtain a better understanding of the actual costs of administering each program and provide the Legislature additional information on each program's true cost-effectiveness. However, because CARB only began using these codes at the beginning of the 2021-22 fiscal year, it has not yet had the opportunity to report on actual costs to the Legislature as we recommended. Therefore, we assess this recommendation as partially implemented.

60-Day Agency Response

CARB has identified the following steps it will take this year to implement the recommendation:

Step 1: Collaborate internally with CARB divisions to identify all transportation programs including establishing lists of those currently accounted for and those that are missing. We have started this step.

Step 2: Process and develop service locations and labor codes. CARB Budgets/Accounting staff will process the templates and develop the codes within FI$Cal and coordinate with divisions to find how these programs should be coded (Reference/Fund/Program). Once established in FI$Cal, work with CARB Human Resources to add the newly established service locations as labor codes in Tempo Timekeeping System.

Step 3: Educate CARB staff on service locations (FI$Cal)/timecodes (Tempo) and how they should be used. This includes: (1) meeting internally with division leadership to discuss the codes and expectations for use from both a labor and an operating expenses and equipment perspective, and (2) meeting with CARB staff that work within transportation programs giving them the newly developed codes and an education about the appropriate uses and vision from their divisional leadership.

Step 4: Send monthly reports sent to Divisions with Transportation Programs. Pull the data monthly and develop reports for submission to divisions. Request feedback on data and track potential gaps with missing information. Develop additional service locations/time codes if necessary based on feedback and program changes.

CARB hopes to have this system in place by the end of 2021.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2020-114

Agency responses received are posted verbatim.