2023-106 University of California

June 6, 2024
2023-106

The Governor of California
President pro Tempore of the Senate
Speaker of the Assembly
State Capitol
Sacramento, California 95814

Dear Governor and Legislative Leaders:

As directed by the Joint Legislative Audit Committee, my office conducted an audit of the University of California (UC) and its use of online program management firms (OPMs). The following report details the audit’s findings and conclusions. Overall, we determined that UC makes limited use of OPM firms to provide online education but should provide increased oversight to better ensure transparency for students.

We identified 51 UC contracts with OPMs in effect as of January 1, 2023, none of which involved undergraduate education. Of those contracts, 30 were with the five UC campuses we selected for review—Berkeley, Davis, Los Angeles, San Diego, and Santa Barbara—of which 15 pertained to continuing education provided through campuses’ extension units. We found that each of these campuses provided potential students with incomplete or misleading information about the OPMs’ involvement in some programs—such as not disclosing whether the OPM was teaching the class—or overstating the value of those programs. Additionally, we found that some of the campus extension units did not consistently adhere to their processes for approving courses or instructors.

The campuses lack systemwide guidance from the UC Office of the President (Office of the President) on contracting with OPMs. For example, although the Office of the President recommends that campuses not engage in incentive compensation—such as tuition revenue sharing—when contracting with third parties to recruit undergraduate students, this guidance does not address graduate or continuing education students. Further, some of the contracts included payment terms that may elevate the risk of OPMs using practices to recruit and enroll students that are not in the best interests of students. To mitigate the risks of using OPMs, the Office of the President should establish systemwide guidance that ensures transparency about OPMs’ involvement in educational programs, enables campuses to provide adequate oversight of OPMs, and protects prospective students from potentially abusive recruiting practices.

Respectfully submitted,

Grant Parks
California State Auditor

Selected Abbreviations Used in This Report

BEE	UC Berkeley Executive Education
ED	U.S. Department of Education
GAO	U.S. Government Accountability Office
OPM	online program manager
UC	University of California
WASC	Western Association of Schools and Colleges

Summary

Online courses and programs have become increasingly common in higher education. Many colleges work with third-party vendors known as online program managers (OPMs), which assist in the development and implementation of online programs. OPMs generally provide instruction and support services, such as marketing, recruiting, course development, and technology-related support. In this audit, we examined the University of California’s (UC) use of OPMs at five campuses—University of California, Berkeley (UC Berkeley); University of California, Davis (UC Davis); University of California, Los Angeles (UCLA); University of California, San Diego (UC San Diego); and University of California, Santa Barbara (UC Santa Barbara)—and drew the following conclusions:

Introduction

Background

Enrollment in online education has increased significantly over the past decade. According to the National Center for Education Statistics, the proportion of college students nationwide enrolled in online education courses increased from about 26 percent of total enrollment in 2012 to nearly 59 percent in 2021. During the COVID-19 pandemic, the use of online education increased in 2020 to more than 73 percent. Although it has since decreased from pandemic levels, online education continues to play a significant role in higher education.

The UC’s governance comprises several entities. The text box describes the system’s governance structure. UC provides education through three divisions: undergraduate, graduate, and continuing education offered through extension units. The UC undergraduate and graduate education divisions provide degree programs to enrolled UC students, while extension units provide a variety of courses to a broader student population. Before enrolling in courses, undergraduate and graduate programs require students to go through an admissions process that includes minimum admission requirements established by the Academic Senate. In contrast, extension units use open enrollment, which permits students to register for UC courses without going through an application and admission process. Extension units provide some courses that offer credit that students may apply toward a degree. The UC Office of the President (Office of the President) indicates that extension units do not receive any ongoing support from the State’s General Fund, and the Board of Regents of the University of California (Board of Regents) exempts nondegree extension unit courses from the Academic Senate’s oversight of courses and curricula.

UC Online Education

UC teaches undergraduate, graduate, and continuing education students in person at its 10 campuses, and it also provides students with online instruction. Undergraduate degree programs may include online courses, although their use is currently limited compared to graduate and extension unit programs. However, in December 2023, the UC President approved the establishment of a presidential task force to assess methods of delivering instruction and to consider criteria for online bachelor’s degree programs of a quality expected from UC. Later, in February 2024, the Board of Regents declined to approve an Academic Senate regulation that would have established a minimum requirement of instructional hours in person and on campus for an undergraduate student to be eligible for a bachelor’s degree. The proposed regulation would have essentially prohibited the offering of a fully online undergraduate degree. In contrast, as of January 2024, UC does offer several fully online graduate programs that culminate in graduate degrees.

Further, UC provides continuing education online through extension units at nine of its 10 campuses. UC San Francisco is the only campus that does not have an extension unit. These extension units are administratively self-contained and operate independently from campuses under systemwide policies. This independence permits the extension units’ open enrollment model, through which all potential students have access to continuing education that is intended to enhance their skills.

Online Program Managers

Like many other universities and colleges, UC contracts with OPMs to support its online programs with services such as marketing, recruiting, course development, technology-related support, and instruction. In general, colleges may contract with more than one OPM for different services or to support different online education programs. The U.S. Government Accountability Office (GAO) reported that as of July 2021, at least 550 colleges nationwide were working with an OPM to support at least 2,900 online education programs. OPMs provide support to a range of education programs, including undergraduate degree programs, graduate degree programs, intensive skill‑based programs such as boot camps, continuing and professional education programs, and public online courses—also known as massive open online courses. Such offerings are open enrollment online courses, meaning that any member of the public may enroll in them, and they are developed by the universities or OPMs and hosted on the OPMs’ websites. According to the GAO, OPMs receive payment for their services through a share of tuition revenue, a predetermined fee, or both.

The Joint Legislative Audit Committee (Audit Committee) directed our office to determine the extent of partnerships between OPMs and a selection of five UC campuses, the level of transparency provided to prospective students, the quality of instruction provided, the level of student satisfaction with the associated courses or programs, student outcomes, and compliance with federal and state laws. For this audit, we examined OPM use at UC Berkeley, UC Davis, UCLA, UC San Diego, and UC Santa Barbara, which we selected based on enrollment, the extent of campuses’ online programming, and public information about the campuses’ use of OPMs.

Issues

UC Uses OPMs to Teach Students in Some Nondegree Programs but Is Not Always Transparent About Doing So

UC Extension Units Have Not Provided Consistent Oversight of OPM Instruction

Campuses Lack Certain Guidance From the Office of the President on Contracting With OPMs

Information on UC extension units’ OPM program websites about program costs appeared to be accurate, but most program websites included minimal information about refund policies. The websites for four of the campus extension units describe a $30 to $40 drop or refund processing fee that generally applies to all extension unit courses. However, when we asked program representatives of four of the campuses’ technology boot camps about the refund policies, they informed us of a separate nonrefundable $1,000 deposit. The program websites made only passing reference to such a deposit and did not clearly disclose its nonrefundable nature or clarify the amount. As of May 2024, we had not observed changes to the websites. The program representatives stated that students must pay this $1,000 nonrefundable deposit even if they drop the program within a week of the first class. Awareness of this nonrefundable deposit might affect prospective students’ interest in these programs, especially those students of limited means. Nearly 20 percent of respondents to our survey who disclosed their annual individual income earned less than $29,000 in the year before enrolling in their program, and 67 respondents indicated that they obtained private loans to pay for the costs of their programs. Without disclosure of the amount of a nonrefundable deposit, prospective students cannot fully consider the financial commitment of enrolling in a program.

Use of Potentially Misleading Terminology

Some UC campuses and their extension units use terminology that could mislead students about the academic value of educational programs. Specifically, some OPM programs describe certificates or awards of completion using terms that sound similar to academic terminology but do not have the same meaning because they do not confer academic credit. For example, UC San Diego contracts with an OPM that uses the term MicroMasters for an open enrollment data science program that does not result in a master’s degree nor provide credit toward a master’s degree at UC San Diego. Because the term master’s generally denotes a graduate degree, terminology of this nature could be confusing to students. In fact, in a 2019 report on microcredentials, a task force of the UC Berkeley division of the Academic Senate raised concerns about misleading terminology, including this term. UC San Diego’s associate vice chancellor of educational innovation stated that its division of the Academic Senate has also expressed concerns about the term. However, as of May 2024, the program website still used this terminology.

Compounding the problem of using this misleading phrase, a UC San Diego representative for the MicroMasters program misrepresented its value when we inquired about it in the guise of a prospective student. He stated that although the program was not required for applications to UC San Diego’s online master’s program in data science, it was beneficial to the admissions process. However, according to both UC San Diego’s director of digital learning and its associate vice chancellor, an applicant’s completion of the MicroMasters program is not taken into account during the admissions process for UC San Diego’s master’s program in data science. The director of digital learning stated that the campus would review the website and speak with program representatives to ensure that information provided does not suggest a connection between the MicroMasters and master’s programs. She later stated that the program representative may have given incorrect information because the campus had not adequately communicated details about the differences between the two programs to all involved parties. Such omissions in communication could lead prospective students to believe that there were additional benefits, such as positively influencing the decision for admission to a degree program, that do not actually exist.

Lack of Defined Review Processes

Campuses typically have the authority to control how their programs are publicly presented: 26 of the 30 OPM contracts we reviewed made marketing materials subject to review by campus personnel. However, only one of the campuses we reviewed—UC Davis—could provide us with a written policy or procedure describing a structured process for reviewing marketing materials or website content. Davis Extension’s documented procedures apply primarily to the marketing strategy for launching a new program, although its senior director of strategic partnerships noted that the unit reviews web pages as necessary throughout the year and as part of its annual planning process. In contrast, UC San Diego’s director of digital learning stated that UC San Diego does not have a formal mechanism for overseeing the marketing or website content created by its OPMs, although the associate vice chancellor for innovation said the campus should adopt a formal process for reviewing marketing and website content. A senior international and business development manager for UC Santa Barbara Professional and Continuing Education (Santa Barbara Extension) stated that although the extension unit and the OPMs agree on initial strategy for marketing, the OPM may later make changes to web content directly without the unit’s approval. Despite two of its three contracts with OPMs requiring that marketing content be subject to Santa Barbara Extension’s approval, the unit does not routinely review website content and is thus unlikely to approve every item of content.

In response to our requests, campuses were able to provide some type of evidence that their staff had reviewed and approved certain content for most of the programs we assessed. However, some campuses were more thorough than others. For example, the UC Davis Graduate School of Management has created a tool for reviewing the messaging for its online MBA program that included tracking changes for the related website’s content. In contrast, although Berkeley Extension developed the marketing and web content for its paralegal studies certificate program, the unit could not provide any evidence that it had reviewed that content. The website stated that 82 percent of graduates surveyed would recommend this certificate. However, when we inquired about the statistic, Berkeley Extension determined that the statistic resulted from a graduate survey conducted in 2017, and the unit could not produce the survey when we requested it.

If Berkeley Extension had a periodic review process for its program websites, it might have identified the potentially outdated statistic and the need to reassess the opinions of paralegal studies graduates. By contrast, when we discussed with UC Davis the inaccurate citation of its graduate program’s rankings, its leadership was already aware of the issue and was in the process of addressing it with its OPM partner. These contrasting examples demonstrate that a thorough review process that identifies and tracks all of the website elements that need periodic review can be an effective method for ensuring that potential students have accurate information when determining whether to pursue particular courses.

More than 60 percent of respondents to our survey indicated that campuses had misrepresented information about an OPM program in some manner. In particular, 136 respondents indicated that they felt their programs misrepresented the employability of graduates, and this factor seemed particularly important because nearly 70 percent of respondents answered that they enrolled in a program for the purpose of changing their career or field of work. However, about 67 percent of survey respondents indicated that the program or instruction they received did not impact their job or employment situation, and only 36 percent of survey respondents agreed that the program was worth the money they invested.

Many students who responded to our survey indicated that they chose to enroll at the extension unit at their specific UC campus instead of at another educational institution or provider because of the reputation or quality of the UC campus and the value that a certificate from a UC campus represents on their resumes. Encountering misleading or false information about OPM programs would likely tarnish students’ perception of the UC. More significantly, UC’s misleading or false information about OPM programs could lead students to invest their time and money in programs that will not provide the benefits advertised on UC or OPM websites. The shortcomings in the marketing information for OPM programs to potential students, along with responses to our survey, suggest that the campuses need additional guidance about how the programs should be marketed. As of May 2024, the Office of the President had not begun the process of sharing systemwide guidance about how transparent campuses should be regarding OPMs’ involvement in these programs. The Office of the President’s executive advisor for academic planning and policy development stated that the office is already drafting possible guidance and intends to start the process of vetting that guidance with stakeholders immediately following the publication of this audit report.

Santa Barbara Extension’s Process for Approving OPM Courses Does Not Assess Instructor Qualifications

Academic Senate regulations require that the relevant university department and the dean of the applicable extension unit approve courses that provide credit toward an academic degree or toward a professional credential or certificate. However, Academic Senate regulations do not require such an approval process for courses that do not meet these criteria, such as most technology boot camps.

Each campus had a process to approve OPM courses, instructors, or both. Staff at each of the extension units noted that their process for reviewing courses included examining the proposed course curriculum and other documentation before approving the course. Four of the five campuses we reviewed evaluated an instructor’s qualifications by reviewing a resume or other biographical information and some additional steps, such as obtaining references, conducting interviews, and observing the proposed instructor conduct a mock lesson. Because they review and approve instructors, the extension units at UC Berkeley, UC Davis, UCLA, and UC San Diego provide more assurance that their OPM instructors are qualified to teach the courses that OPMs offer through UC. Only Santa Barbara Extension did not have a process to approve OPM instructors.

According to Santa Barbara Extension’s senior international and business development manager, the unit does not see a need to oversee OPM instructors because the unit trusts its OPMs to ensure quality instruction. However, other extension units have identified concerns when reviewing the qualifications of prospective instructors. For example, Davis Extension’s senior director of strategic partnerships described an instance in which a potential instructor did not demonstrate the preferred level of confidence in teaching, nor did the instructor’s resume show substantial teaching experience. Davis Extension moved forward with approving the instructor after determining that the instructor exhibited strengths in teaching small groups and that the OPM would provide her with training and support. Without such a review process, students taking courses through the Santa Barbara Extension may not receive the quality of instruction they expect from the UC and for which they paid.

Santa Barbara Extension’s executive director said that the COVID-19 pandemic drove it to expand online course offerings and that student demand drove the unit to provide courses on coding through partnerships with other entities, such as OPMs. However, Santa Barbara Extension indicated that imposing additional processes, such as reviewing instructors’ qualifications, could potentially impact the economic benefits of using OPMs. Nevertheless, we found that such a process does not appear to be excessively time consuming or costly. For example, Davis Extension’s senior director of strategic partnerships said that the unit’s process for reviewing an instructor candidate’s application and qualifications usually takes 90 minutes to two hours if a candidate meets the unit’s requirements.

Some Extension Units Have Not Consistently Followed Their Own Processes for Overseeing OPM Courses and Instructors

Each of the five extension units we reviewed had processes in place to approve OPM courses, and four of them had processes in place to approve OPM instructors. However, the units must follow all of their processes to ensure effectiveness. We reviewed 21 courses in total, selecting at least four courses that each extension unit offered in 2022. Three of the extension units we reviewed were able to demonstrate that they consistently followed every step of their processes for approving OPM courses. Only one of the four extension units with an instructor review process could demonstrate that it followed each process step for all of the courses we reviewed. As Table 2 shows, only Davis Extension was able to provide evidence that it followed each step of the process it had established for reviewing both courses and instructors. Santa Barbara Extension followed its established course approval process, but the unit did not have an established instructor review process, as we note earlier. For eight of the 13 courses that we reviewed at the other three extension units, the units either did not follow all of the steps in the processes they had established for reviewing instructors, or were unable to demonstrate that they followed each step. UCLA Extension and Berkeley Extension were able to provide documented approvals for all or most of the OPM courses reviewed, respectively. However, each of the three units was unable to demonstrate that it approved an instructor of at least one of the courses reviewed. Berkeley Extension was unable to provide supporting documentation, such as instructor references, for any of the four courses reviewed. Although San Diego Extension has a policy that describes its approval process, for three of its five courses we reviewed, it could not demonstrate its approval of the courses or the instructors. Because the extension units rely on OPMs to provide courses, extension units’ approvals of courses and instructors are a key safeguard to ensure that courses and instructors meet UC standards and that students receive the quality of instruction they expect.

The three extension units attributed these lapses to staff turnover or poor document retention practices. For example, for one of its boot camps, San Diego Extension’s assistant dean stated that the unit may have overlooked the instructor and course approval because the approval was pending when the assigned program manager departed and he may not have communicated the approval status to the incoming manager. The assistant dean said that to address this issue, San Diego Extension will examine its process to identify possible improvements so that all of its stakeholders know whether the course and instructor have been approved. One of the courses at San Diego Extension that did not have documented approval of the course or instructor was a course conferring credit toward a professional certificate and was subject to Academic Senate regulations that require certain approvals. In this case, the lack of approval for the course puts San Diego Extension out of compliance with Academic Senate regulations, in addition to being noncompliant with its own processes.

Similarly, UCLA Extension’s assistant dean of academic affairs explained that he could not provide evidence that the extension unit followed its process because a previous program director had departed abruptly, leading to the course and instructor approval documents not being properly transferred to the new program director. He said that UCLA Extension has since improved its recordkeeping practices by storing in the unit’s central database all documents related to course and instructor approvals for OPMs, and this practice should allow other staff to access the documents even if program managers leave the unit.

Berkeley Extension could not demonstrate that it followed its established instructor approval process in several instances because it did not retain approval documentation. The contracts we reviewed for Berkeley Extension have been in place for multiple years, and the unit approved the four courses we reviewed before 2019. The assistant dean stated that the unit has since adopted a stronger record retention process and now retains electronically all documents related to course and instructor approvals. By deviating from established processes and regulations for the approval of OPM courses and instructors, UC risks offering students courses that do not meet UC standards and diminishing the value of such courses to students.

Certain Extension Units Did Not Use Student Course Evaluations to Monitor OPM Instruction

The Western Association of Schools and Colleges’ (WASC) Senior College and University Commission has established guidance on agreements with unaccredited entities such as OPMs.⁴ That guidance states that the accredited institution should, among other things, establish procedures for periodically evaluating the efficacy and quality of the unaccredited entity’s services and the outcomes the entity provides. A 2022 report in the International Journal of Teaching and Learning in Higher Education notes that course evaluations are commonly used to measure teaching effectiveness at universities.

At all extension units that we reviewed, staff indicated that students’ course evaluations occurred as part of the typical process for extension unit courses. However, campuses varied in their approach to student course evaluations for OPM courses. San Diego Extension used its own standard course and instructor survey for the OPM courses that we reviewed. In contrast, Davis Extension’s dean explained that its process for student course evaluations varies by OPM and that one OPM performs its own surveys and evaluations, while the unit administers its own course evaluations for the other OPM programs using the same format it follows for other extension unit courses. Berkeley Extension’s program director of business and management stated that one of its OPM partners did not administer student course evaluations and that another OPM manages its own student course evaluations and provides summaries to the campus. The program director also said that Berkeley Extension has begun to implement its own student course evaluations for all of its boot camp courses.

Both Santa Barbara and UCLA extension units stated that their respective OPMs conduct student course evaluations but that the units do not receive regular reports from the OPMs on the evaluations. Santa Barbara Extension’s senior international and business development manager explained that staff meet with OPMs and sometimes discuss these evaluations, but they do not perform a detailed review of the OPMs’ evaluation data. UCLA Extension’s assistant dean of academic affairs acknowledged that the unit does not regularly review the results of the evaluations conducted by the OPMs, although the OPM occasionally shares its results upon request from the extension unit or as needed. Staff at both Santa Barbara and UCLA extension units stated that OPMs have an incentive to ensure student satisfaction and the quality of education they provide to protect their reputations. However, because an OPM’s interests may differ from those of a campus, the unit’s reliance on the OPMs to notify it of concerns may be misplaced. Santa Barbara Extension’s senior international and business development manager acknowledged that there are risks to not performing a detailed review of the evaluations administered by the OPMs but stated that the extension unit trusts the OPMs to inform it if there are problems. UCLA Extension’s assistant dean of academic affairs stated that the extension unit ensures that students of OPM courses know that they have access to the same feedback systems as other students do to file complaints about any problems they experience.

Our survey responses suggest that some students have concerns with courses taught by OPMs that the campuses should address. For example, 111 of 321 survey respondents who answered a question about the coaching, career support services, and other services provided through their OPM programs indicated that they were dissatisfied.⁵ The text box includes a selection of the students’ responses. In response to another survey question, 216 of 322 students reported that the program or instruction they received did not impact their job situation at all. The potential impact of the course they took on their employment seemed important to many students because 224 of 324 respondents reported that they enrolled in the program they took for the purpose of changing their career or field of work. Because the extension units did not routinely use OPMs’ evaluations to monitor students’ experiences and satisfaction, the units lacked a proven means of identifying ineffective courses and instructors and were consequently less able to take proactive steps to improve outcomes for students.

Other Areas We Reviewed

During the course of our audit, we identified concerns pertaining to updates to federal incentive compensation guidance and the revenue amounts Berkeley Extension received from an OPM. We also reviewed student completion rates for OPM-instructed education programs and the processes campuses used when entering into OPM contracts.

Updates to U.S. Department of Education Incentive Compensation Guidance

The Audit Committee directed us to review the selected campuses’ compliance with the federal ban on incentive compensation and, to the extent possible, determine how UC can better demonstrate its compliance. As we describe earlier in this report, federal law does not generally allow institutions, including UC, to provide financial incentives that are based on success in recruitment activities, unless the U.S. Department of Education’s (ED) bundled services exception applies. When we contacted the ED, the representative with whom we spoke did not describe concerns about UC specifically but did convey general concerns about colleges’ ability to demonstrate compliance with the federal prohibition on incentive compensation. The representative said that a good practice that a college should use to prevent incentive compensation is to include in its contracts a provision that its annual compliance auditor must have access to the contractor’s records. Further, the college should require the OPM to have an independent evaluation of incentive compensation performed annually, either as part of a broad compliance audit requirement or through a separate requirement. However, compliance audits are less helpful when the criteria for allowable activities are unclear, such as ambiguity about when the bundled services exception applies. When we asked the ED representative whether a bonus payment from a campus to an OPM in a tuition revenue-sharing agreement would be compliant with the ban on incentive compensation, which was a situation we encountered during the audit, he declined to provide an opinion.

Additional federal guidance for auditing incentive compensation may be forthcoming. The GAO recommended in 2022 that the ED provide more instructions about OPM arrangements for consideration during federal compliance audits and program reviews. The Office of Management and Budget addressed this GAO report in May 2023 with guidance for auditors, suggesting that they should obtain lists of entities that recruit students for colleges and review those entities’ compensation records. The ED appears to be evaluating incentive compensation rules more broadly. In February 2023, the ED issued guidance stating that certain regulated organizations that contract with institutions, known as third-party servicers, would include entities that provide recruitment and retention services for colleges. This guidance would subject entities performing these functions to additional audit requirements for third-party servicers. The ED initially set an effective date of September 1, 2023, for this guidance to be followed. However, the ED published a letter in May 2023 that effectively rescinded the previous guidance. Nevertheless, the ED held sessions in March 2023 to receive public comments, recommendations, and suggestions to improve guidance on incentive compensation law, particularly with respect to bundled services. These sessions may lead to clarifying information from the ED that will inform UC’s approach to ensuring compliance with the law. In the May 2023 letter, the ED stated that it plans to release final revised guidance at a later date.

Concerns About the OPM Revenue That Berkeley Extension Received

As we discuss in our assessment of data reliability, we noted concerns in the financial data related to Berkeley Extension’s contract with one OPM. Among our concerns, we found that the OPM may not have paid sufficient revenue to Berkeley Extension. Berkeley Extension’s contract with this OPM states that 20 percent of the gross revenue collected should be distributed to the campus. However, we found that for fiscal year 2021–22, the revenue actually received by Berkeley Extension was a 14 percent share of the total receipts reported by the OPM. We were not able to calculate a precise amount for the potential revenue Berkeley Extension may not have received due to timing issues associated with student payment plans. Berkeley Extension amended the contract with the OPM to change the revenue share terms from 20 percent of the gross revenue collected to 20 percent of the net revenue collected, after deducting refunds, discounts, and defaults. This amendment, however, did not become effective until May 1, 2023, after the end of fiscal year 2021–22.

According to the Berkeley Extension accounting manager, the accounting department does not have a way to independently verify the amount of tuition charged nor the receipts collected from students. Further, the accounting manager stated that the accounting department assumed that the Berkeley Extension academic department involved with the OPM reviewed the OPM’s financial information for accuracy. However, the Berkeley Extension program director for that academic department stated that the academic department did not have a process to verify that the revenue amounts received from the OPM were accurate. Without a means to ensure that the OPM pays Berkeley Extension the full amounts owed according to the terms of the contract, there is heightened risk that the campus is not receiving its fair share of tuition revenue.

Student Completion Rates for Programs That Involved OPM Instruction

The Audit Committee directed us to review the completion and dropout rates for courses taught by OPM-hired instructors. We analyzed enrollment data for each of the UC extension units at the selected campuses to identify the percentage of students who withdrew, completed and passed, or completed but did not pass a course or program taught by an OPM-hired instructor from July 2020 through December 2022. As Table 3 shows, we compared these rates to the overall rates for all courses and programs at each extension unit, including those that provide academic credit.

We found that overall completion rates varied across the extension units, in part because of the different tracking and reporting methods that each unit used. For example, Santa Barbara Extension’s senior international and business development manager explained that grades are not required for the unit’s 800-level courses, which are not credit bearing, resulting in a greater number of blank grades and a higher percentage in the “Other” category. Additionally, we found that the completion rate for San Diego Extension’s technology boot camps, which we calculated as 100 percent, could be overstated. Specifically, San Diego Extension tracks only the number of students who complete the program, but it does not keep a record of those who drop out or withdraw. Further, in our analysis, we considered a withdrawal to be a drop that occurred after the refund deadline. However, Davis Extension does not distinguish between course drops occurring before and course withdrawals occurring after the refund deadline. Therefore, we could not identify withdrawals from the data Davis Extension provided. As a result of the limitations we found, the data we show in Table 3 may not accurately present completion rates for all of the courses we reviewed.

Processes Campuses Used When Entering Into OPM Contracts

Campuses generally did not have established processes for contracting with OPMs. Instead, they each followed standard UC procedures for contracting. UC Davis and UC Berkeley also have their own procedures: UC Davis has its own procurement policy that is not specific to OPMs, which is based on the Office of the President’s policy, the State Contracting Manual, and state law. UC Berkeley now has a policy specific to OPMs, established in September 2022, which specifically requires competitive bidding for revenue-sharing OPM agreements, among other requirements. The Office of the President’s systemwide procurement policy for the purchases of goods and services requires competitive bidding for purchase agreements of more than $100,000 unless certain exceptions apply. The policy describes formal competitive bidding as the process of publicly issuing documents that invite vendors to submit offers to fulfill specific requirements.

We did not determine that any of the contracts we identified violated UC contracting requirements. However, as part of our review, we determined whether the campuses we selected considered multiple vendors when entering into OPM contracts, such as through a competitive bidding process. We found that the campuses used a competitive bidding process for only one of the 30 OPM contracts we reviewed.

The campuses provided various reasons for not using a competitive bidding process when entering into OPM contracts. For 20 of the 30 contracts we reviewed, the campuses asserted that it was not necessary to consider multiple vendors because the contracts include tuition sharing payment terms or represented no cost to the campus. Specifically, they stated that the Office of the President’s procurement policy did not apply to these contracts because the policy specifically covers contracts involving an expenditure of funds, not those that are revenue generating. Further, the campuses stated that, as part of their research for entering in the OPM contracts, they researched other vendors when appropriate. The campuses did not document much of this. However, given that these contracts are revenue generating, the campuses’ explanations are reasonable.

The Office of the President’s procurement policy also contains several exceptions to the competitive bidding requirement. For example, the requirement does not apply to contracts of less than $100,000 annually, which the campuses cited as justification for not using a competitive process for four of the OPM contracts we reviewed. Similarly, UC Davis cited an exception that applies when the requested services provided by a particular source are the only ones that will meet the campus’s needs. In this instance, UC Davis procurement documents indicated that it did consider multiple vendors but determined that only one OPM demonstrated the ability to meet all of the program’s needs. In three other instances, UCLA contracted for OPM services by selecting vendors from a prequalified pool of vendors it had previously established. This is a common practice throughout the State.

Recommendations

Office of the President

To promote practices that will mitigate the risks of using OPMs, the Office of the President should, with input from relevant stakeholders, such as extension unit deans and using the guidance provided by WASC, create guidance by June 2025 for UC’s use of OPMs. The guidance should define the OPMs to which it applies and, at a minimum, establish the following expectations:

• Campuses should ensure a minimum level of transparency about an OPM’s involvement in education programs, including requiring accurate public‑facing website descriptions of the services an OPM provides and requiring the publication of the names and qualifications of OPM-paid instructors on public‑facing websites, when applicable. This guidance should establish expectations for the minimum level of review that campuses should perform of websites and other marketing or recruiting materials for their programs that involve an OPM.

• Campuses should provide adequate oversight of OPMs that provide instruction on behalf of UC and the steps the campuses should take to review, approve, and evaluate OPM-created courses and OPM-provided instructors. The guidance should specifically address collecting student evaluations of OPM courses and should establish expectations for campuses’ reviews of those evaluations following each academic term.

To better protect prospective students from recruiting practices that are not in their best interests, the Office of the President should expand its existing guidance on incentive compensation by June 2025 to address graduate and continuing education students. The expanded guidance should describe that UC discourages tuition revenue sharing or bonus payments to entities that recruit graduate or continuing education students, as it does for undergraduate students. Further, the expanded guidance should describe the safeguards that campuses should adopt to mitigate the risks posed to students and to better ensure UC’s compliance with federal law.

UC Berkeley, UC Davis, UCLA, UC San Diego, UC Santa Barbara

To provide transparency to prospective students regarding course instruction, by June 2025, the campuses should engage with relevant stakeholders regarding creating or amending their policies or processes to require that course descriptions and program websites include the following:

• Disclosure of the partnership between an OPM and the campus.
• A description of the OPM’s roles, particularly when the OPM provides instruction.
• Identification of OPM instructors as well as their training and experience.
• Disclosure of the amounts of any nonrefundable deposits or fees.

To ensure that they do not provide misleading information on their websites, by June 2025, the campuses should engage with relevant stakeholders regarding creating and implementing a policy to review, on no less than an annual basis, the program websites associated with courses that OPMs support to ensure that the information provided is current and accurate. The campuses should formally document these reviews and track any needed or subsequent changes.

UC Berkeley

To ensure that Berkeley Extension receives the full amounts owed to the campus according to the terms of its OPM contracts, by June 2025, Berkeley Extension should establish a process to verify that the revenue amounts received from each OPM are accurate.

UCLA and UC Santa Barbara

To better monitor how well the OPMs are serving students, by June 2025, UCLA Extension and Santa Barbara Extension should establish and implement a policy to review and assess the results of OPM-administered student course evaluations after each academic term.

UC San Diego

To prevent prospective students from being misled about the value of OPM‑provided courses, by June 2025, UC San Diego should develop fact sheets that list key information for each course or program, such as whether the course provides a benefit for admission to another campus degree program.

To prevent prospective students from being misled about the value of OPM-provided courses, by June 2025, UC San Diego should request that its Academic Senate assess whether the campus should continue to use the term MicroMasters.

To comply with Academic Senate rules and its own policies regarding course and instructor approvals, by June 2025, San Diego Extension should establish a process to document all OPM course and instructor approvals.

UC Santa Barbara

To ensure that the OPM instructors teaching students on its behalf are sufficiently qualified, by June 2025, Santa Barbara Extension should incorporate a review of the instructors’ qualifications into its OPM course approval process.

We conducted this performance audit in accordance with generally accepted government auditing standards and under the authority vested in the California State Auditor by Government Code section 8543 et seq. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on the audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.
Respectfully submitted,

Grant Parks
California State Auditor

June 6, 2024

Staff:
John Lewis, MPA, CIA, Audit Principal
Nicole Madera, MPP, Senior Auditor
Bonnie Roy, PhD, Senior Auditor
Rachel Adams
Maggie Carroll
Alyssa Centeno
Eliana Flores, CPA

Legal Counsel:
David King

Appendices

Appendix A—Contract Information for Selected Campuses

Appendix B—Tuition Revenue Divided Between Campuses and OPMs in
Fiscal Year 2021–22

Appendix C—Student Costs for Programs That Involved OPM Instruction

Appendix D—Results of Our Survey of UC Students in OPM-Instructed
Extension Unit Courses

Appendix E—Scope and Methodology

Appendix A

Contract Information for Selected Campuses

The Audit Committee directed us to identify and review the agreements in effect as of January 1, 2023, that five selected UC campuses and the Office of the President had entered into with OPMs. In addition, the Audit Committee directed us to identify which contracts required OPM employees to provide direct instruction to students; determine whether contracts included incentive compensation to OPM providers based on student enrollment; evaluate whether contracts required an OPM to disclose to students its contracted relationship with UC and whether the instructor was a UC employee; determine the roles played by UC, the instructors, and the OPMs under the contract; and determine how the contracts divided revenue between the university and the OPM. As part of that review, we also assessed the length of contract terms and grounds for termination; terms relating to marketing and misrepresentation of the educational programs under the contract; and certain academic provisions under the contract, such as whether the course was credit bearing. Table A1, Table A2, Table A3, and Table A4 summarize the results of these reviews.

---

Appendix B

Tuition Revenue Divided Between Campuses and OPMs in Fiscal Year 2021–22

The Audit Committee directed us to identify how campuses and OPMs divided the funds from contracts that involved OPM instruction. Of the 30 contracts we reviewed, 15 required the OPM to provide instruction. These 15 contracts all related to nondegree programs. Table A3 and Table B summarize how campuses and OPMs divided the revenue resulting from the contracts.

During our review, we identified that a portion of the revenue generated from OPM programs came from student financial aid sources, including both private loans and state and federal grants. For example, students collectively spent about $84,000 in state funding to pay for their enrollments in a Santa Barbara technology boot camp. These students received funding in the form of supplemental job displacement vouchers that the Department of Industrial Relations awarded them. In another example, students in a Berkeley Extension technology boot camp received private loans totaling about $725,000 to pay for the program.

Overall, we determined that students spent around $3.2 million in private and public financial aid to help pay for tuition in fiscal year 2021–22 related to the 15 instruction contracts. Of this amount, only about $260,000 was public funding, which is less than 1 percent of the total revenue the contracts generated for that period. We identified the student financial aid sources through the campus-provided financial reports. However, some students may have paid for OPM programs with funding sources that were not reported by the campuses. Because of this limitation in the data, the amount we identified may not represent the total amount of public or private loans used for student tuition.

Table B presents the total tuition revenue that the programs associated with these contracts generated and the amount of revenue that campuses and OPMs received, respectively, in fiscal year 2021–22.

---

Appendix C

Student Costs for Programs That Involved OPM Instruction

The Audit Committee directed us to determine how much campuses charged students for courses and programs provided through OPM contracts. Because contracts did not consistently disclose cost information for these programs, with few exceptions we had to identify tuition costs from information that was posted on program websites during our review in the summer and fall of 2023. In addition, we could not identify costs for some programs—for example, we could not identify costs when a campus did not specify tuition amounts in a contract or other documentation and was not offering the program during the time we performed our review. Tuition costs ranged from a low of less than $400 to a high of nearly $14,500. OPM programs offering instruction in synchronous (live online) class sessions over a period of several months, similar to one or more traditional academic terms, were generally the most expensive programs.

Table C presents the tuition costs we identified for the five selected campuses. Programs for which we could not identify tuition costs are not presented in the table.

---

Appendix D

Results of Our Survey of UC Students in OPM-Instructed Extension Unit Courses

The Audit Committee directed us to conduct a survey of students who received instruction from OPM-paid instructors to determine the following: students’ demographic information, whether students felt that marketing and recruiting efforts were transparent and appropriate, how students paid for the relevant tuition, whether students were satisfied with the instruction provided, whether students felt the course was worthwhile and assisted them in their careers, and whether students believed their instruction came from the university. Of the 3,320 UC students who enrolled in an OPM-instructed course at one of the selected campuses in 2022, 338 responded to our survey. Table D1 summarizes the number of students we surveyed and the number of students who responded from each of the selected campuses. Slightly more than 10 percent of students to whom we sent the survey provided responses. We cannot draw broad conclusions from the results because of the relatively low response rate. Moreover, there exists the possibility that a larger portion of the responses were from students who were displeased with their experiences, and the survey received a mixture of positive and negative responses. Notwithstanding these limitations, the responses of those students who did participate suggest that there is room for improvement.

The majority of survey respondents indicated that they enrolled in a program to change their employment situation, yet only 33 percent believed that the course they enrolled in had an impact on their job situation, as Figure D1 shows. Although nearly half of the respondents felt that the quality of instruction met their expectations for their UC campus, many felt that the program was not worth the money they invested, as Figure D2 illustrates.

Figure D1

Source: Responses to California State Auditor survey.

The first pie chart shows the result of the survey question, “Did you enroll for the purpose of changing your career or field of work?” 69 percent of respondents answered “Yes” and 31 percent of respondents answered “No.”

The second pie chart shows the result of the survey question, “How did the program impact your employment?” 67 percent of respondents answered “Did not impact job situation,” 15 percent answered “Improvement in current job,” 13 percent answered “Found a job at a new company/organization,” and 5 percent answered “New role at the company.”

Figure D2

Source: Responses to California State Auditor survey.

The first pie chart shows the result of the survey statement, “The quality of instruction met my expectations.” 47 percent of respondents indicated that they agreed with this statement, 18 percent neither agreed nor disagreed, and 35 percent disagreed with the statement.

The second pie chart shows the result of the survey statement, “The program was worth the money I invested.” 35 percent agreed with this statement, 16 percent neither agreed nor disagreed, and 49 percent disagreed with the statement.

Survey responses suggest that campuses should do more to improve transparency. For example, the majority of students who responded to the survey were not aware that the instruction for their program was provided by an employee of a third party and not by university faculty, as Figure D3 shows. Table D2, which describes each survey question we asked, presents results for a number of different questions that indicate many students felt that one or more aspects of the program they participated in was misrepresented to them. Finally, Table D3 summarizes the results of the optional demographic questions we asked.

Figure D3

Source: Responses to California State Auditor survey.

The pie chart shows the result of the survey question, “Were you aware that your online instruction was being provided by an employee of a third party?” 55 percent of respondents answered “No” and 45 percent of respondents answered “Yes.”

---

Appendix E

Scope and Methodology

The Audit Committee directed us to conduct an audit of UC’s use of OPMs to determine the extent of these partnerships, the level of transparency provided to prospective students, the oversight provided, the level of student satisfaction with the associated courses or programs, the outcomes that students achieved, and the programs’ compliance with federal and state laws. Table E lists the objectives that the Audit Committee approved and the methods we used to address them. Unless otherwise stated in the table or elsewhere in the report, statements and conclusions about items selected for review should not be projected to the population.

Berkeley Executive Education

UC Berkeley Executive Education (BEE) is a separate entity from UC Berkeley. However, BEE has a contract with UC Berkeley to provide executive education programs under the UC Berkeley brand. Under that agreement, BEE also provides online executive education for which it has three OPM agreements. These programs provided in partnership with OPMs could pose a risk of misrepresentation because they deliver education under the UC Berkeley brand.

Therefore, we performed limited work to ensure that BEE was not delegating instruction to OPMs and misrepresenting it as provided by UC Berkeley faculty. We reviewed BEE’s OPM agreements, analyzed program information on its website, and spoke to a program representative while portraying the role of a prospective student for one of BEE’s OPM programs. We determined that UC Berkeley employees were providing instruction for the BEE programs that use OPMs. We further determined that for the program we reviewed, BEE disclosed the OPM partnership to prospective students and generally provided information that was consistent with the contract provisions we reviewed. However, for one BEE OPM agreement we reviewed, the relevant program webpages did not include a disclaimer that distinguishes between UC Berkeley and BEE that aligns with the affiliation agreement between UC Berkeley and BEE. We discussed this issue with UC Berkeley, indicating that it should consider clarifying the distinction between UC Berkeley and BEE on the program website. The dean of UC Berkeley’s Haas School of Business stated that she reached out to BEE, which performed an audit of its webpages and is working with its site developer to correct any program webpages that do not include the disclaimer. She also stated that UC Berkeley has added language to its support center and terms-of-use sections of the BEE website, clarifying that BEE is a California nonprofit organization formed for the purpose of operating executive education programs in support of UC Berkeley and the Haas School of Business. We subsequently observed that this text had been added to the stated sections of the BEE website.

Assessment of Data Reliability

The U.S. Government Accountability Office, whose standards we are statutorily obligated to follow, requires us to assess the sufficiency and appropriateness of the computer‑processed information that we use to materially support our findings, conclusions, or recommendations. In the course of this audit, we relied on the data and testing that we describe below.

To determine the amount of revenue generated from OPM contracts for the period of July 1, 2021, through June 30, 2022, we obtained electronic data files from the selected campuses’ financial reporting systems. To evaluate the data, we interviewed staff knowledgeable about the data. We performed dataset verification procedures and electronic testing of key data elements and did not identify any issues. We also performed accuracy testing by comparing the revenue shares actually collected for fiscal year 2021–22 to the payment terms found in the related contracts, as well as by obtaining supporting documentation for any student loan amounts included in the student tuition payments. We did not identify any issues resulting from these procedures.

We also compared the total revenue in each UC campus’s financial system reports to OPM reports that included detailed enrollment information to determine whether the totals agreed. We then traced the detailed enrollment information from the OPM reports to enrollment information obtained from the registrar for each campus’s extension unit. We found the data to be materially complete for all OPM contracts except for the UC Berkeley contract with one OPM, which we discussed above. We found several exceptions in which this OPM’s enrollment data included transactions that the data obtained from the registrar omitted. Thus, we could not conclude that the OPM’s contract data are sufficiently complete for our purposes. Although this determination may affect the precision of the numbers we present, there is sufficient evidence in total to support our findings, conclusions, and recommendations. For all other selected campuses, we found the data sufficiently reliable for the purposes of determining the amount of tuition that campuses charged to students enrolling in OPM-instruction programs, the amount of revenue the campus and the OPM received, and any costs incurred by the campus for fiscal year 2021–22.

To answer various audit objectives, we obtained electronic data files from the enrollment systems of the selected UC extension units for the period from July 1, 2020, through March 31, 2023. Four of the selected extension units (Berkeley Extension, Davis Extension, UCLA Extension, and Santa Barbara Extension) use the Destiny One System to process enrollment records, and San Diego Extension uses a system called ISIS. To evaluate the available data, we interviewed staff knowledgeable about the data and performed dataset verification procedures and electronic testing of key data elements, and we did not find any issues material to our analysis. To assess the completeness and accuracy of the respective campuses’ enrollment data, we compared them to data about the certificates issued to students who completed extension unit courses. Berkeley Extension, Davis Extension, UCLA Extension, and Santa Barbara Extension each use a third-party entity to issue certificates, while San Diego Extension issues its own certificates. We determined that UCLA Extension’s enrollment data were sufficiently reliable for our purposes. We also determined that San Diego Extension’s enrollment data were sufficiently reliable, with the exception of courses provided by one OPM whose students were never entered into the unit’s enrollment system.⁶

For Davis Extension and Santa Barbara Extension, we found differences between the enrollment data and respective certificate systems, and thus we determined that the enrollment data were not complete. Further, because of the concerns described above regarding Berkeley Extension’s contract with one OPM, we concluded that Berkeley Extension’s enrollment data were not complete. However, we determined that the enrollment data provided by the selected extension units were the best available data for our purposes. Because of the various errors we identified, it is possible that our analysis excluded some students who were enrolled during the period we reviewed. Although these errors may affect the precision of the numbers we present, there is sufficient evidence in total to support our findings, conclusions, and recommendations, and these errors did not affect other aspects of our analysis.

Responses

University of California, Office of the President

University of California, Berkeley

University of California, Davis

University of California, Los Angeles

California State Auditor’s Comment on the Response from the University of California, Los Angeles

University of California, San Diego

University of California, Santa Barbara

---

---

---

---