Report 2014-602 All Recommendation Responses
Report 2014-602: High Risk Update—California Department of Technology: Lack of Guidance, Potentially Conflicting Roles, and Staffing Issues Continue to Make Oversight of State Information Technology Projects High Risk (Release Date: March 2015)
Recommendation #1 To: Technology, California Department of
By December 2015 CalTech should develop and adopt criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction, including when and how IPO analysts should recommend corrective action and escalate issues to CalTech's management.
Annual Follow-Up Agency Response From November 2017
Reflected in SIMM 45, CDT developed and adopted criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction.
- Completion Date: March 2017
California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented
Annual Follow-Up Agency Response From October 2016
ITPOD has developed a draft escalation and correction action plan. The final version is expected to be completed in December 2016.
- Estimated Completion Date: December 2016
California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented
1-Year Agency Response
The Information Technology Project Oversight Division ITPOD has developed an escalation process which includes a reporting template and examples of conditions that warrant escalation (See attached). In the course of drafting the escalation process, ITPOD decided to also develop a "Corrective Action Plan" process which will formally notify departments that certain conditions or negative trends must be remediated, and if allowed to continue, could result in suspension or termination of the project. This process will become official State policy which necessitates allowing for consultation with Agency Information Officers and drafting State Administrative Manual updates. The additional time needed for this recommendation is due to the anticipated time needed to appropriately articulate and vet the conditions which warrant this action.
- Estimated Completion Date: July 2016
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Partially Implemented
6-Month Agency Response
Activity is progressing in developing the criteria for future use in determining project intervention.
- Estimated Completion Date: Dec 2015
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
60-Day Agency Response
- Communicate the criteria for suspending or terminating a project to ITPOD - complete
- Develop high level conditions which can be observed and documented - in process
- Document ITPOD's internal escalation process - in process
- Estimated Completion Date: 12/31/2015
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: Pending
Although CalTech provided us documentation that it met with its staff to communicate its criteria for suspending or terminating a project, the criteria provided was the same as that included in its response to our report, which we indicated was not sufficient. Specifically, we noted that the criteria provided in its response was a good start; however, the criteria only speaks to circumstances where the need to intervene is apparent. We believe guidance is important in situations where professional judgment is needed—as we note on page 13 of our report—such as before IT projects reach the situations CalTech describes. Furthermore, CalTech's response does not address our concerns discussed on page 15 of our report that, beyond the description in state law, it has not defined the remedial measures, short of suspension or termination, that it may pursue, or the fact that CalTech lacks criteria to guide independent project oversight (IPO) analysts in recommending such measures. According to our IT expert, without such guidance, CalTech cannot meaningfully defend its decisions about whether and when to intervene in a troubled IT project.
Recommendation #2 To: Technology, California Department of
By December 2015 CalTech should develop and adopt criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction, including when and what CalTech should require that sponsoring agencies perform as remedial actions, and what sanctions CalTech will impose for noncompliance with these remedial actions.
Annual Follow-Up Agency Response From November 2017
Reflected in SIMM 45, CDT developed and adopted criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction.
- Completion Date: March 2017
California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented
Annual Follow-Up Agency Response From October 2016
ITPOD has developed a draft escalation and correction action plan. The final version is expected to be completed in December 2016.
- Estimated Completion Date: December 2016
California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented
1-Year Agency Response
The Information Technology Project Oversight Division ITPOD has developed an escalation process which includes a reporting template and examples of conditions that warrant escalation (See attached). In the course of drafting the escalation process, ITPOD decided to also develop a "Corrective Action Plan" process which will formally notify departments that certain conditions or negative trends must be remediated, and if allowed to continue, could result in suspension or termination of the project. This process will become official State policy which necessitates allowing for consultation with Agency Information Officers and drafting State Administrative Manual updates. The additional time needed for this recommendation is due to the anticipated time needed to appropriately articulate and vet the conditions which warrant this action.
- Estimated Completion Date: July 2016
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Partially Implemented
6-Month Agency Response
Activity is progressing in developing the criteria for future use in determining project intervention.
- Estimated Completion Date: Dec 2015
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
60-Day Agency Response
- Develop a "progressive discipline" framework to communicate the need for corrective action to departments - in process
- Estimated Completion Date: 13/31/2015
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: Pending
Recommendation #3 To: Technology, California Department of
By December 2015 CalTech should develop and adopt criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction, including what conditions could trigger CalTech to consider suspending or terminating an IT project.
Annual Follow-Up Agency Response From November 2017
Reflected in SIMM 45, CDT developed and adopted criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction.
- Completion Date: March 2017
California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented
Annual Follow-Up Agency Response From October 2016
ITPOD has developed a draft escalation and correction action plan. The final version is expected to be completed in December 2016.
- Estimated Completion Date: December 2016
California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented
1-Year Agency Response
The Information Technology Project Oversight Division ITPOD has developed an escalation process which includes a reporting template and examples of conditions that warrant escalation (See attached). In the course of drafting the escalation process, ITPOD decided to also develop a "Corrective Action Plan" process which will formally notify departments that certain conditions or negative trends must be remediated, and if allowed to continue, could result in suspension or termination of the project. This process will become official State policy which necessitates allowing for consultation with Agency Information Officers and drafting State Administrative Manual updates. The additional time needed for this recommendation is due to the anticipated time needed to appropriately articulate and vet the conditions which warrant this action.
- Estimated Completion Date: July 2016
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Partially Implemented
6-Month Agency Response
Activity is progressing in developing the criteria for future use in determining project intervention.
- Estimated Completion Date: Dec 2015
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
60-Day Agency Response
- Draft policy for SAM or SIMM - not started
- Incorporate all content into ITPOD training program - not started
- Estimated Completion Date: 12/31/2015
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: No Action Taken
Recommendation #4 To: Technology, California Department of
To clarify and reinforce its oversight authority with sponsoring agencies, by December 2015 CalTech should develop a method to formally document and communicate its expectations with the sponsoring agencies whose projects are under CalTech's oversight.
Annual Follow-Up Agency Response From November 2017
The new SIMM 45, with a project oversight fact sheet was developed and posted.
- Completion Date: April 2017
California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented
CalTech provided us its revised IT Project Oversight Fact Sheet, which more clearly outlines its expectations for departments when performing oversight on IT projects. These expectations clearly state that IPO analysts will have access to all documents and meetings for an IT project, and that departments are to take appropriate actions to respond to recommendations that IPO analysts make. Therefore, we consider this recommendation fully implemented.
Annual Follow-Up Agency Response From October 2016
ITPOD has developed a draft "Project Oversight Fact Sheet", which will be referenced in and enclosed with the approval letter for exiting each stage of the Project Approval Lifecycle. The final version is expected to be completed in December 2016.
- Estimated Completion Date: December 2016
California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented
1-Year Agency Response
ITPOD has developed a draft "Project Oversight Fact Sheet" which will be referenced in and enclosed with the approval letter for exiting each stage of the Project Approval Lifecycle. (See attached draft Fact Sheet and S1BA Approval Letter Template)
- Completion Date: March 2016
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Partially Implemented
While Caltech has spelled its general framework for authority, it needs to include in this framework the importance of it being independent when performing independent project oversight.
- Auditee did not address all aspects of the recommendation
6-Month Agency Response
Activity is progressing in developing an expectation document for agencies.
- Estimated Completion Date: Dec 2015
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
60-Day Agency Response
Plan for Implementation:
- Develop a script for ITPO managers to use at the beginning of each stage of the project approval process which defines the purpose of the stage, the roles and responsibilities of CalTech and the state entity during the stage and the high level conditions for completing the stage - In process
- Develop an approval template to be used at the end of Stage 2 and Stage 4 which includes a citation of CalTech's authority to oversee projects - Not started
- Estimated Completion Date: 12/31/2015
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: Pending
Recommendation #5 To: Technology, California Department of
To help ensure the independence and objectivity of IPO analysts working in the oversight and consulting division, CalTech should develop a policy outlining expectations for independence and objectivity while performing oversight of IT projects.
Annual Follow-Up Agency Response From October 2016
In working with California State Auditor (CSA), ITPOD had completed an updated version of the Principles and Expectations document with additional content on the definition and subject of "independence". This updated document will be forwarded to CSA for closure of this recommendation.
- Completion Date: October 2016
California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented
1-Year Agency Response
ITPOD has established a Principles & Expectations document for all staff. (See attachment) Among a number of important expectations, this document states that ITPO managers must understand and maintain their independence while performing project oversight work. This is now provided to every new staff member and signed by both the staff person and their manager. ITPOD has also included the topic of "independence" as a standing agenda item in our regular weekly staff meetings on Friday. (See attachment) We encourage the staff to share their successes and challenges in maintaining their independence.
- Completion Date: November 2015
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Partially Implemented
Caltech's "Principles & Expectations" document lacks a definition of independence, and it does not address how IPO analysts will maintain their independence while performing oversight of IT projects.
- Auditee did not address all aspects of the recommendation
6-Month Agency Response
Activity is progressing in developing a new policy of expectations for IPO analysts.
- Estimated Completion Date: Dec 2015
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
60-Day Agency Response
- Change the name of the IT Project Oversight and Consulting Division (ITPOC) to IT Project Oversight Division (ITPOD) - partially complete
- Develop a policy for ITPOD staff to follow - in process
- Add the concept of independence as a standing item for discussion on the ITPOD staff meeting agenda - completed
- Incorporate the independence concept into the scripts for each stage referenced above - not started
- Estimated Completion Date: 12/31/2015
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: Partially Implemented
Recommendation #6 To: Technology, California Department of
To help ensure the independence and objectivity of IPO analysts working in the oversight and consulting division, CalTech should provide regular training regarding maintaining independence while conducting project oversight.
Annual Follow-Up Agency Response From October 2019
The CDT Office of Statewide Project Delivery (OSPD) has developed and implemented a four-hour curriculum and scheduled and delivered training to the Project Approvals and Oversight staff and now requires this training to be refreshed annually as part of annual training requirements.
- Completion Date: September 2019
California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented
Annual Follow-Up Agency Response From October 2018
CDT has developed procedures and training to have new and current IPO analysis take regular training involving independence. Independence training was last provided on Oct 5, 2018, to IPO analysts, supervisors and managers.
This independence training is planned to be conducted at least annually.
- Completion Date: October 2018
California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented
We reviewed the October 2018 training as well as information CalTech provided in previous years for this recommendation. The slides for this training presentation focused on background information about independence from the IEEE standards and Government Code, which we acknowledge are important to understand the IPO analyst's role and authority. However, as we stated in our response in 2017, we indicated this training should provide examples of situations and coaching to staff. We added this explanation in response to CalTech's request for clarification of what actions we were looking for it to take to address this recommendation. We also note that CalTech's "principles and expectations" document, which it provided in 2016, does include an appropriate definition of independence, but the training should teach IPO analysts how to operationalize this independence through a demonstration of situational problems that occur while IPO analysts are serving in their oversight roles. Finally, the October 2018 training was part of an overall meeting that included five other topics and was scheduled to last for 90 minutes, which indicates that independence was not the primary topic and time spent on it was unclear.
- Auditee did not substantiate its claim of full implementation
Annual Follow-Up Agency Response From November 2017
The Quarterly Training Plan needs to be updated to cover Independence including providing examples of situations and coaching to staff. Expected completion date is March 2018.
- Estimated Completion Date: March 2018
California State Auditor's Assessment of Annual Follow-Up Status: Pending
Annual Follow-Up Agency Response From October 2016
In working with California State Auditor (CSA), ITPOD had completed an updated version of the Principles and Expectations document with additional content on the definition and subject of "independence". This updated document will be forwarded to CSA for closure of this recommendation.
- Completion Date: October 2016
California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented
Although CalTech indicates that it includes the topic of independence at each weekly staff meeting and has adopted policy on independence, we believe that IPO analysts need regular training on independence since it is a critical and challenging aspect of them providing independent project oversight on IT projects.
- Auditee did not address all aspects of the recommendation
1-Year Agency Response
ITPOD has established a Principles & Expectations document for all staff. (See attachment) Among a number of important expectations, this document states that ITPO managers must understand and maintain their independence while performing project oversight work. This is now provided to every new staff member and signed by both the staff person and their manager. ITPOD has also included the topic of "independence" as a standing agenda item in our regular weekly staff meetings on Friday. (See attachment) We encourage the staff to share their successes and challenges in maintaining their independence.
- Completion Date: November 2015
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Partially Implemented
As noted for recommendation 5, the "Principles & Expectations" document lacks a definition for independence while performing IT project oversight.
6-Month Agency Response
Activity is progressing in developing a new training program for IPO analysts.
- Estimated Completion Date: Dec 2015
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
60-Day Agency Response
- Incorporate the independence policy into the new employee orientation module of the training program - not started
- Estimated Completion Date: 12/31/2015
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: No Action Taken
Recommendation #7 To: Technology, California Department of
To better track its oversight actions and sponsoring agencies' responses to these actions, CalTech should retain the briefing documents created for the State CIO's portfolio meetings and the project status reports that sponsoring agencies submit while project oversight is ongoing.
Annual Follow-Up Agency Response From October 2016
In working with CSA, ITPOD successfully developed a new folder system in SharePoint to capture and retain each month's Briefing Document for all projects subject to CDT oversight. Screen copies of this new folder system and a sample of non-confidential information will be forwarded to CSA this month for closure of this recommendation.
- Completion Date: October 2016
California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented
1-Year Agency Response
Briefing Documents are being saved in ITPOD Share Point system. Copies of saved briefing documents and recorded action items from portfolio meetings were previously provided to the auditors as evidence of completion. (Recent examples are attached)
a. The expectation that every staff member must maintain and update the briefing document is included in an ITPOD Principles and Expectations document.
- Completion Date: July 2015
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Partially Implemented
Caltech's response does not address the retention of project status reports that agencies submit while project oversight is ongoing.
- Auditee did not address all aspects of the recommendation
6-Month Agency Response
ITPOD staff are saving the briefing documents and recording action items from portfolio meetings in the SharePoint e-File room.
- Completion Date: July 2015
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
During our follow up with CalTech to obtain documentation to support its response, CalTech informed us that its e-File room in SharePoint needs to be modified to implement version control so that when a new briefing document is uploaded it does not overwrite the prior version. Also, CalTech believes that its PPM Central System that is used to publish project status reports on its public website has a bug that may affect the presentation of the reports' information. As a result, CalTech is still working to fully implement our recommendation to retain briefing documents and project status reports.
- Auditee did not substantiate its claim of full implementation
60-Day Agency Response
Plan for Implementation:
- Develop a written process for ITPOD staff to follow to retain all briefing documents and project status reports while the projects are in an active status - in process
- Develop a location in SharePoint for retention of action items from Portfolio meetings - not started
- Estimated Completion Date: 07/31/2015
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: Pending
Recommendation #8 To: Technology, California Department of
To better track its oversight actions and sponsoring agencies' responses to these actions, CalTech should record action items from all portfolio meetings.
1-Year Agency Response
Briefing Documents are being saved in ITPOD Share Point system. Copies of saved briefing documents and recorded action items from portfolio meetings were previously provided to the auditors as evidence of completion. (Recent examples are attached)
b. ITPOD established a process of recording action items from portfolio meetings.
- Completion Date: July 2015
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Fully Implemented
6-Month Agency Response
ITPOD staff are saving the briefing documents and recording action items from portfolio meetings in the SharePoint e-File room.
- Completion Date: July 2015
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
During our follow up with CalTech to obtain documentation to support its response, CalTech informed us that its e-File room in SharePoint needs to be modified to implement version control so that when a new briefing document is uploaded it does not overwrite the prior version. As a result, CalTech is still working to fully implement our recommendation.
- Auditee did not substantiate its claim of full implementation
60-Day Agency Response
- Develop a process for capturing action items from Portfolio meetings and sharing these with the ITPOD manager for follow-up and retention - not started
- Estimated Completion Date: 07/31/2015
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: No Action Taken
Recommendation #9 To: Technology, California Department of
To ensure that the sponsoring agencies' project status reports provide a reliable and consistent assessment of an IT project's progress, CalTech should develop and adopt specific standards that describe how to calculate and report the project's current status.
Annual Follow-Up Agency Response From October 2018
The Project Status Report has been finalized and was included in an update to SIMM 45 and associated Technology letter (TL 2017-07) in December 2017.
https://cdt.ca.gov/wp-content/uploads/2017/12/SIMM45F_-2017-1205.pdf
- Completion Date: December 2017
California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented
With the new project status report in place, CalTech will now be in a better position to critically review and evaluate the status of IT projects that departments report to it.
Annual Follow-Up Agency Response From November 2017
The revised Project Status Report has been drafted and it is expected that this will be included in an update to SIMM 45 and an associated Technology letter in December 2017.
- Estimated Completion Date: December 2017
California State Auditor's Assessment of Annual Follow-Up Status: Pending
Annual Follow-Up Agency Response From October 2016
ITPOD has drafted a new Project Status Report format, which includes specific instructions for measuring and reporting of project performance that is related to the schedule, scope and cost. Effort is still ongoing with the Agency Chief Information Officers in implementing this new process and publishing this new policy in the State Administrative Manual.
- Estimated Completion Date: April 2017
California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented
1-Year Agency Response
Completion of this recommendation was delayed while the new CA Project Management Framework was being developed. That effort is nearly complete and CDT recently discussed the concerns related to the CA-PMM instructions for completing project status reports with the Agency Information Officers and is considering two options.
a. Create a new Project Status Report format, which includes specific instructions for measuring and reporting of project performance related to schedule, scope and cost. This would be completed by the Project Manager.
b. Utilize the project status dashboard in a new version of the Independent Project Oversight Report (IPOR) which is completed by the ITPOD IPO manager.
The AIOs agreed to work with ITPOD to determine the best approach with the goal of publishing a new policy in SAM by July 2016.
- Estimated Completion Date: July 2016
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Pending
6-Month Agency Response
Activity is progressing in developing and adopting specific standards for project status reports.
- Estimated Completion Date: Dec 2015
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
60-Day Agency Response
Plan for Implementation:
- Determine if PSRs should continue to be required and if not, identify an appropriate alternative to providing public information about the status of all reportable IT Projects - in process
- Develop standards for measuring project progress and the project's status - not started
- Publish the new standards - not started
- Estimated Completion Date: 12/31/2015
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: Pending
Recommendation #10 To: Technology, California Department of
To attract and retain employees with appropriate experience and qualifications to perform IT project oversight, CalTech should continue its efforts to gain approval to modify and use the project manager classification for the IPO analyst role.
Annual Follow-Up Agency Response From October 2018
New IT Classifications rolled out statewide in Feb 2018 and new organization chart and position duty statements were developed and implemented.
- Completion Date: February 2018
California State Auditor's Assessment of Annual Follow-Up Status: Resolved
CalTech now uses the Information Technology Supervisor and Manager series to perform the IPO analyst role. The CalTech duty statements for these positions generally align with the role of the IPO analyst.
Annual Follow-Up Agency Response From November 2017
The administration is revisiting the IT classification category.
- Estimated Completion Date: January 2019
California State Auditor's Assessment of Annual Follow-Up Status: Pending
Annual Follow-Up Agency Response From October 2016
It was determined in October 2015 by CalHR that the existing Project Management classification is not suitable for use by ITPOD. CalHR has taken the lead role to modernize a variety of state classifications including IT classifications, and CDT is an active participant in this reform initiative. As stated previously, until better options become available; ITPOD will continue to use the Data Processing Manager (DPM) classification and work with CalHR to gain approval to use the project management classification.
- Estimated Completion Date: June 2017
California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented
1-Year Agency Response
It was determined in October 2015 that the existing Project Management (PM) classification is not suitable for use by ITPOD. Because the audit recommendation was based on the assumption that the existing PM classification would assist ITPOD in improving staff retention, we ask that this recommendation be classified as completed. That said, CalHR is leading a significant effort to modernize a variety of state classifications including IT classifications. CDT is an active participant in this reform initiative. Until better options become available, ITPOD will continue to use the Data Processing Manager (DPM) classification.
- Completion Date: October 2015
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Pending
6-Month Agency Response
Activity is progressing and a determination regarding the use of PM classification is expected to be made by or before December 2015.
- Estimated Completion Date: Dec 2015
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
60-Day Agency Response
Plan for Implementation:
- Evaluate the current classification of PM to determine suitability for ITPOD - in process
- Identify any needed changes and determine if these can be made to the existing PM classification or if a new classification will be required - in process
- Work with CalHR to confirm findings and evaluate options - not started
- Obtain CalHR approval of the best option - not started
- Estimated Completion Date: 12/31/2015
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: Pending
Recommendation #11 To: Technology, California Department of
To ensure that it provides the appropriate level of oversight for IT projects under development, by December 2015 CalTech should conduct a workload assessment to determine the level of staffing and expertise required for each IT project it oversees.
1-Year Agency Response
ITPOD has established three processes to capture and evaluate workload information:
a) Stage Gate Tracking Spreadsheet (attached) - this provides the status and expectations for completing the work in each stage of the Project Approval Lifecycle. It will also provide earlier insight as to what level and when oversight work will be needed when the project is approved.
b) ITPOD established a monthly report of work activity by activity type, by manager and by branch (attached). This provides Branch managers and the Deputy Director with information about current capacity of individuals and groups within the Division. It will also facilitate better planning for future staff augmentation needs.
c) Effective April 1, every IPO manager will develop a work plan with their supervisor. (attached) This plan will cover a period to be determined by the supervisor depending on the experience and performance of the IPO manager. The work plan will include all anticipated work and relative priorities and, at the end of the period, will document accomplishments and explain any variances to the plan.
- Completion Date: March 2016
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Fully Implemented
6-Month Agency Response
Activity is progressing in developing the workload assessment.
- Estimated Completion Date: Dec 2015
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
60-Day Agency Response
Plan for Implementation:
- Capture quarterly updates of workload activity by ITPOC manager and by Branch - in process
- Determine what new projects will begin in 2015-16 - completed
- Request additional staff needed for 2015-16 projects through provisional budget language - in process
- Develop a BCP for 2016-17 staff augmentation accounting for new projects and other workload activity in the justification - in process
- Document a process for on-going workload tracking and workload projections - not started
- Estimated Completion Date: 12/31/2015
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: Pending
Recommendation #12 To: Technology, California Department of
Using the workload assessment that we recommended CalTech complete by December 2015, CalTech should make decisions to assign staffing to oversee each IT project. This staffing could include contracted IPO consultants in those situations when CalTech staff either are unavailable or lack the expertise needed.
1-Year Agency Response
ITPOD has established three processes to capture and evaluate workload information:
a) Stage Gate Tracking Spreadsheet (attached) - this provides the status and expectations for completing the work in each stage of the Project Approval Lifecycle. It will also provide earlier insight as to what level and when oversight work will be needed when the project is approved.
b) ITPOD established a monthly report of work activity by activity type, by manager and by branch (attached). This provides Branch managers and the Deputy Director with information about current capacity of individuals and groups within the Division. It will also facilitate better planning for future staff augmentation needs.
c) Effective April 1, every IPO manager will develop a work plan with their supervisor. (attached) This plan will cover a period to be determined by the supervisor depending on the experience and performance of the IPO manager. The work plan will include all anticipated work and relative priorities and, at the end of the period, will document accomplishments and explain any variances to the plan.
- Completion Date: March 2016
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Fully Implemented
6-Month Agency Response
Activity is progressing in developing the workload assessment.
- Estimated Completion Date: Dec 2015
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
60-Day Agency Response
Once Caltech has completed the workload assessment, and receives a decision for its staffing resource request, it will be in a better position to make decisions to assign the appropriate staffing resources.
- Estimated Completion Date: 02/29/2016
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: No Action Taken
Recommendation #13 To: Technology, California Department of
To improve its oversight training, by June 2015 CalTech should continue to implement a consistent and repeatable training plan for IPO analysts, which includes contract management, project assessment, IT systems engineering, and maintaining independence.
Annual Follow-Up Agency Response From October 2018
In addition to the mandated Annual Training Plans for each IPO analysts relating to state-wide project management courses, CDT has developed an IPO specific self-assessment training needs plan for IPO analysts. This plan covers the required IPO analysts' duties including the four areas identified by the state auditor.
- Completion Date: October 2018
California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented
We evaluated as fully implemented because CalTech has a training plan in place. However, we did not evaluate the adequacy of training areas that it included in the training plan.
Annual Follow-Up Agency Response From November 2017
The new onboarding process training has been developed. IT Projects Quarterly Training Plan will be updated to cover areas identified by the California State Auditor (CSA). Estimated completion date is June 2018.
- Estimated Completion Date: June 2018
California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented
Annual Follow-Up Agency Response From October 2016
In conjunction with CSA'a advice, the ITPOD's plan for implementation includes:
- Develop training priorities for new staff and for continuing staff-in process.
- Identify existing training courses that can be leverage-in process.
- Develop customized content for any additional training course that are needed.
- Finalize curriculum and schedule.
- Seek additional funding, if needed.
- Estimated Completion Date: June 2017
California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented
1-Year Agency Response
As of April 2016, ITPOD will began using a new employee onboarding checklist which will ensure every new IPO manager is provided with the information to perform their work functions and which will be used to plan for the training they will need. (See attachment)
- Completion Date: March 2016
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Partially Implemented
The information CalTech provided does not address how IPO analysts will be trained on an ongoing basis, such as the extent of and timing of required ongoing training and continuing education.
- Auditee did not address all aspects of the recommendation
6-Month Agency Response
Workload and vacancies have impeded progress on the development of this plan. Additional time will be needed to complete it.
- Estimated Completion Date: Jan 2016
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
60-Day Agency Response
Plan for Implementation:
- Develop training priorities for new staff and for continuing staff - in process
- Identify existing training courses that can be leveraged - in process
- Develop customized content for any additional training courses that are needed - not started
- Finalize curriculum and schedule - not started
- Seek additional funding if needed - not started
- Estimated Completion Date: 06/30/2015
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: Pending
Recommendation #14 To: Technology, California Department of
By June 2015 CalTech should put in place a system to track IPO analysts' training hours to ensure that all IPO analysts have completed the necessary California Project Management Methodology training curriculum.
Annual Follow-Up Agency Response From October 2016
The CA-PMF replaced the CA-PMM policy in June 2016. The CalQ requirement was not included in the new policy. The individual courses that were previously included in the CalQ curriculum have been included in the ITPOD Staff Competency Checklist. This checklist will be forwarded to CSA this month for closing out this recommendation.
- Completion Date: October 2016
California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented
The combination of CalTech's two actions--implementation of software in January 2016 to track staff training and adoption of a curriculum policy in June 2016--satisfies our recommendation.
1-Year Agency Response
ITPOD managers now have ready access to training courses taken by ITPOD staff. In January 2016, the CDT Office of Professional Development implemented a major PeopleSoft upgrade to the Training Registration System (TRS). The new system was renamed the Learning Request System (LRS). This system provides several new benefits:
o Improved user-friendly dashboard and screen environment
o Integrated access to the Training & Education Center Catalog and automated enrollment for TEC classes
o Allow administrators to mass-enroll employees into required courses such as Driver Training, Sexual Harassment Prevention, and Security Training
o Reminders feature—staff can easily see approved upcoming training; requested training is easily visible in manager's console
o Upgraded email notification system
In June 2016, the CA Project Management Manual (CA-PMM) will be replaced by the CA Project Management Framework (CA-PMF). The CDT CA Project Management Office (CA-PMO) will be providing CA-PMF training on two levels: for those beginning to learn project management practices and methodologies and for those who would benefit from more in-depth practice in developing tools and templates. The former CalQ requirement will be replaced with a new policy which will recognize experience and credentials from other legitimate sources. ITPOD will take advantage of these opportunities as they become available.
- Estimated Completion Date: June 2016
- Response Date: March 2016
California State Auditor's Assessment of 1-Year Status: Partially Implemented
6-Month Agency Response
Workload & vacancies have impeded progress to implement a tracking system for staff training. Additional time will be needed to complete it.
- Estimated Completion Date: Jan 2016
- Response Date: September 2015
California State Auditor's Assessment of 6-Month Status: Pending
60-Day Agency Response
Plan for Implementation:
- Collect information on staff completion of CA-PMM training - in process
- Maintain the information in a Division wide spreadsheet - in process
- Estimated Completion Date: 06/30/2015
- Response Date: May 2015
California State Auditor's Assessment of 60-Day Status: Pending
All Recommendations in 2014-602
Agency responses received are posted verbatim.