This Web page has been archived on the Web.

2008 December Report of the Auditor General of Canada Chapter 5—Managing Information Technology Investments—Canada Revenue Agency

2008 December Report of the Auditor General of Canada

Chapter 5—Managing Information Technology Investments—Canada Revenue Agency

Main Points

Introduction

Focus of the audit

Observations and Recommendations

Making the right investments

Management practices for information technology investments have improved significantly
A more strategic approach is needed to manage IT investments
A more comprehensive performance-reporting framework is needed

Managing information technology projects

The Agency has not complied with its own guidance for managing information technology projects
Only two of the eight audited projects met all four criteria
Project review procedures need to be strengthened

Conclusion

About the Audit

Appendix—List of recommendations

Exhibits:

5.1—Steps have been added to the project management process to improve oversight

5.2—IT projects selected for audit

5.3—Six of eight projects did not meet all criteria

Case Studies:

5.1—Business Intelligence Decision Support

5.2—Compliance System Redesign

5.3—Integrated Charities System

5.4—T2 Two-Dimensional Bar Coding Project

Main Points

What we examined

The Canada Revenue Agency collects some $346 billion in taxes annually on behalf of the Government of Canada, the provinces (except Quebec), the territories, and certain First Nations governments. Processing up to 3.0 million computer transactions per hour, the Agency maintains some of the largest databases in the government and spends about $500 million annually on information technology (IT) systems, of which $129 million is recovered from the Canada Border Services Agency. It is currently pursuing a complex set of strategies to transform its business, including increasing the interactive nature of its systems to improve both its own administration and its relations with taxpayers.

We examined whether its systems and practices provide the Agency with reasonable assurance that IT investments are well managed and consistent with its business objectives. We looked at the management framework for IT investments, including the processes for deciding which IT-enabled business projects to invest in and for monitoring progress to ensure that the investments continue to support the Agency's objectives. We examined eight projects to determine whether they had appropriate governance and accountability structures, a comprehensive business case, and adequate management of risk. We also looked at whether expected benefits from the projects were clearly defined, adequately tracked, and properly reported.

Why it's important

The Canada Revenue Agency's information technology systems are critical to its ability to administer taxes, benefits, and related programs and to ensure compliance with federal, provincial, and territorial tax laws. An organization as large and complex as the Agency needs to ensure that it invests in the right IT systems and applications and that its investments deliver the intended value. Its systems are also the main vehicle the Agency has for improving the efficiency and cost-effectiveness of its tax administration activities, and for improving client and taxpayer services.

What we found

The Agency has responded. The Agency agrees with all of our recommendations. Its detailed responses follow each recommendation throughout the chapter.

Introduction

5.1 The Canada Revenue Agency (CRA) is one of the largest federal government organizations. Its responsibilities include assessing and collecting federal taxes, such as income taxes, the goods and services tax/harmonized services tax (GST/HST), and other taxes. The Agency's mandate also includes

5.2 In addition, the Canada Revenue Agency provides information technology (IT) services and support to the Canada Border Services Agency (CBSA). After the customs function of the Canada Customs and Revenue Agency (CCRA) was transferred to CBSA, the two agencies have continued to share a common network and infrastructure. Under a Memorandum of Understanding (MOU) between CRA and CBSA, it is CRA's responsibility to operate and maintain the network and infrastructure.

5.3 CRA relies heavily on its information technology systems to administer the federal and provincial income tax acts, deliver benefits programs to Canadians, and meet its other clients' needs. Over the years, except for two major incidents that affected electronic filing and accessibility to account information, those systems have been reliable.

5.4 According to its records, in the 2007–08 fiscal year, CRA spent about $509 million on IT activities and employed about 4,000 IT professionals. CRA used about $175 million to develop new systems and to maintain or upgrade existing systems. The remaining $334 million was used to operate existing systems and to provide IT services needed to support CRA business.

5.5 Under the MOU between the two agencies, CRA bills CBSA for the services it provides. In the 2007–08 fiscal year, billings amounted to about $129 million of CRA's total operating costs. The IT Branch in CRA supports an IT infrastructure consisting of 1,200 servers, 6 mainframes, 2 data centres, more than 450 applications, and close to 56,000 desktop computers and laptops.

5.6 The Canada Revenue Agency's demand for IT investments always outstrips available funding, requiring it to balance competing priorities. Those priorities include modernizing existing systems, some of which are more than 20 years old, and developing new systems that are more efficient and reliable or that would better serve taxpayers and benefit recipients. In today's world, delivering individual IT projects on time and within budget is no longer enough. Today's large organizations need management processes that help them to choose their IT investments wisely and to realize the intended value of those investments.

Focus of the audit

5.7 During the audit, we looked at the overall management of the Agency's IT investments and key aspects of project management. We examined the Agency's management systems and practices against widely accepted industry standards, its compliance with its own policies and procedures and with other relevant guidance.

5.8 The audit assessed whether the Agency's systems and practices provide it with reasonable assurance that it is managing its IT investments well, and that they are in line with its business objectives.

5.9 More details on the audit objective, scope, approach, and criteria are in About the Audit at the end of this chapter.

Observations and Recommendations

Making the right investments

Management practices for information technology investments have improved significantly

5.10 We expected the Canada Revenue Agency to have adopted a management framework that included appropriate policies and procedures to provide reasonable assurance that existing information technology investments as well as proposed new ones will support corporate and program priorities.

5.11 We found that the Agency has a sound framework for choosing and managing IT investments. The framework, which has recently been significantly improved, focuses on choosing and managing individual IT projects. To a lesser degree, the framework deals with the management of all the Agency's IT investments.

5.12 In early 2006, after concerns with some large IT projects were identified, the Agency made significant changes to its management practices. We reported on the Agency's concerns about the Integrated Revenue Collection system, in Chapter 8 of our May 2006 Report and Chapter 3 of our November 2006 Report. The changes included the creation of the Resource and Investment Management Committee (RIMC) to provide better oversight of all large projects, including IT, and the development and implementation of an Agency project management policy to better address its own specific needs. In our view, creating RIMC, with its defined roles and responsibilities, is a significant improvement in the oversight of major investment projects, including IT investments.

5.13 To oversee the management of CRA, including its annual IT investments, the Agency employs a framework of senior committees. Its Board of Management oversees the Agency's organization and management. Since December 2006, the Board has been approving plans for any IT projects exceeding $20 million and reviewing the funding and scope of those projects. The Agency has developed a format for project briefings and has started giving the Board of Management quarterly project reports about significant projects.

5.14 The Agency Management Committee determines the Agency's annual budget for IT investments, decides how that budget will be distributed throughout the organization, and selects significant system development projects.

5.15 The following three subcommittees, all of which are chaired by the Commissioner, report to the Agency Management Committee:

5.16 Recently identified improvements. In early 2008, the Resource and Investment Management Committee reviewed the governance structure for overseeing the management of all investments and its own practices for managing IT. The RIMC found that it lacked certain critical information for decision-making and decided the process for monitoring IT projects needed improvement.

5.17 The RIMC recommended that the Agency Management Committee approve a strengthened oversight framework for project approval, planning, and execution to

5.18 Taking these comments into consideration, the Agency's Management Committee approved a more rigorous process for approving and monitoring new project proposals.

5.19 The previous project management framework required only four steps; the new framework requires six steps (Exhibit 5.1). Managers must now satisfy the requirements of two steps before incurring significant costs to develop a full business case. The new project-approval process also includes one step requiring project sponsors to identify the expected benefits and, after implementation, confirm that they have realized these benefits. These changes, once fully implemented, should improve the monitoring of IT development projects by the RIMC.

Exhibit 5.1—Steps have been added to the project management process to improve oversight

Flow charts comparing the old and new project management processes for information technology investments

[text version]

Source: Adapted from the Canada Revenue Agency

5.20 This strengthened process was approved by the Agency Management Committee in February 2008. At the time of our audit, it was too early to review the implementation of these changes. Many IT projects take years to complete, and it will take time to determine how well the Agency's new governance framework for managing IT investments is working. In addition, in the coming months, the RIMC Secretariat will need to work hand-in-hand with project sponsors, to issue appropriate guidance, and to provide suitable training that will ensure the consistent and appropriate implementation of the new project management framework.

5.21 The development of a comprehensive project management policy and framework is not enough to ensure that the desired results will be achieved. As described later in this chapter, when we examined four key aspects of project management, we found that established guidance may not always be followed. The Agency needs to take appropriate steps to ensure that this new project management regime is being respected by the responsible project teams.

5.22 Recommendation. Within two to three years, the Agency Management Committee should ensure that it receives and reviews information on how well its new project management policies, procedures, and guidelines are being implemented and on how well they are being complied with throughout the Agency.

The Agency's response. Agreed. Within two or three years, the Canada Revenue Agency (CRA) will undertake and complete an assessment of how well the enhanced project approval and monitoring framework is being implemented and complied with across the Agency. This kind of periodic review is performed as a matter of course by the CRA. In fact, such a review led to the above-noted improvements to the project oversight framework, which were approved by the Agency's Management Committee in early 2008. These changes included the development of new training and information products to increase the awareness of the new project management regime, as well as more detailed guidelines and templates to facilitate compliance.

A more strategic approach is needed to manage IT investments

5.23 Large organizations must have management practices in place that ensure they focus on current and planned IT investments that best contribute to meeting their business objectives, with an acceptable degree of risk and at reasonable cost. These management practices, referred to as portfolio management, are widely accepted as best practices for the governance of IT investments. Organizations that use portfolio management practices go beyond making decisions on a project-by-project basis and consider the appropriateness of their portfolio of IT investments as a whole. The practices that organizations use are similar to those used by individuals to manage their investment portfolios.

5.24 During our audit, we examined the Agency's systems and practices that would support portfolio management, including the following four key elements:

5.25 We found that the Agency has some processes and information sources that address elements of portfolio management. In some areas, the Agency needs to complete actions that are already under way; and, in two areas, it needs to supplement existing practices.

5.26 As noted previously, the senior management committee structure includes processes for deciding which projects to select, determining project management risks, and monitoring IT project performance. The Agency has other activities and practices related to portfolio management, including

5.27 Multi-year strategic investment plan. In order to manage its portfolio of IT investments, the Agency requires information on how IT investments will meet the business needs of the future. We expected to find that the Agency documented this information in a long-term, strategic IT investment plan. More specifically, we expected the Agency to have a plan consistent with the recent Treasury Board Policy on Investment Planning—Assets and Acquired Services. That policy is intended to ensure that departments and agencies allocate resources in a manner that clearly supports program outcomes and government priorities. While the Agency is not required to follow this policy, it has recognized the need to be consistent with the practice used by other departments and agencies.

5.28 The policy requires the development of an annual investment plan that includes information on the effects of the proposed investment and information on

5.29 We found that the Agency is developing a multi-year strategic investment plan—a deficiency that the Agency has identified through its own self-assessment. It has acknowledged that "this plan is important to ensure that systems and applications have the capacity to meet current and future requirements, in a timely and cost-effective manner. The plan would also identify sustainable applications, applications in need of renewal and an action plan to address the business needs over a 15-20 year time horizon."

5.30 The IT branch has started to work with some of the operating branches to help them assess their long-term business needs. To date, work with the Legislative Policy and Regulatory Affairs branch has been completed, and the IT Branch is currently working with the Appeals Branch.

5.31 Sustainability and risks of current IT applications. To effectively manage other aspects of its portfolio, we expected the Agency to have information about the sustainability and risks associated with its existing suite of IT applications and the future business needs that are to be addressed through the development of IT solutions. We found that the Agency has much of this information and that it is taking action to acquire the information it currently lacks.

5.32 Having a complete list of all IT applications and knowing the associated risks is essential to managing IT investments as a portfolio. We expected to find a complete list of all current IT applications as well as an analysis of their sustainability and any other specific risks. We found that the Agency has made good progress in compiling a list of the IT applications that are currently in use. The Agency's IT branch divides these applications into three separate categories:

5.33 The Agency has assessed the sustainability and the risks associated with the applications that are managed by the IT Branch. However, this has not been completed with the same level of rigour for the applications being maintained by the headquarters operating branches. During our audit, the Agency was still compiling a complete inventory of local applications and had yet to fully assess the associated risks. Although the financial investment in local applications is small, there are potential risks that are significant from a governance perspective.

5.34 The following text box provides more detail on the sustainability of Agency applications and other risks associated with these applications. It also illustrates the challenges facing the Agency's senior management when making choices about IT investments.

National Applications The Agency has assessed the sustainability of national applications. It recently classified 141 out of the approximately 450 applications currently in use by the IT Branch as poorly sustainable because their database platforms or programming language are being phased out and will no longer be used for new applications. The majority of these applications are rated as "high business priority." Because the renewal of these applications will require a significant investment in financial and human resources, managing this renewal is an area of increasing importance.
Applications Controlled by Headquarters' Branches or Regions The Agency has identified 58 applications that support national programs that are operated by branches other than IT. These applications are generally smaller than those operated by the IT branch; they have evolved over time and the Agency has not assessed their sustainability or their risk profile. Because they are not subject to ongoing IT branch governance, the Agency does not know how well they align with its application standards or if they will need to be replaced by national IT branch applications.
Local Solutions There are more than 730 applications known in the Agency as "local solutions." In response to a recent internal audit, the Agency is currently assessing these local solutions for sustainability and risks to the Agency's operations. Because some may be high risk (for example, those that allow Agency employees to download information from national databases), the Agency needs to ensure that proper controls and accountability are in place. Many of the local solutions are believed to meet needs that national applications do not address, so the Agency is analyzing them to determine what, if any, changes should be made to national applications.

5.35 Investment portfolio categories and objectives. As noted in the previous section, the Agency faces some significant challenges related to the sustainability of its existing IT applications. In addition, the Agency has identified the replacement of one of its data centres as a significant risk to its operations. The Agency has recognized that its current resources may be insufficient to undertake all of the necessary IT investments. Accordingly, it will have to make difficult choices about the timing and prioritization of the investments that it chooses to undertake.

5.36 Portfolio management will help the Agency's senior management make strategic decisions about where to invest its IT resources, by using categories that are meaningful to the Agency, and will give senior management an overview of the composition of its portfolio of investments. For example, the Agency's investment portfolio could be categorized in the following way:

5.37 Organizations that adopt portfolio management as a decision-making tool monitor and review the composition of their portfolio periodically, generally once or twice a year. They want their portfolio to reflect an appropriate balance of investments in each category. In some cases, organizations establish ranges they consider appropriate for each category in the portfolio and assess whether the actual portfolio falls within the established objectives. Regardless of how organizations use portfolio management information, in our view, having this information enhances an organization's ability to make IT investment decisions.

5.38 We found that the Agency's portfolio management information is not formally categorized and does not include established objectives for each category, both of which would help senior management make strategic decisions about IT investments.

5.39 Clear evaluation criteria. The Agency's evaluation criteria need to be clearly defined to help it prioritize and select IT investments within each category of its portfolio. Evaluation criteria would include factors such as how well proposed investments align with the Agency's strategic objectives, what the benefits are for the Agency, and what the overall project risks are. Having clear criteria will lead to more consistent and transparent decision making by the Agency.

5.40 The Agency has recently started asking project teams to identify and rank the reasons why their projects are important. However, we found that because the evaluation criteria are not clearly documented, it is not always possible to demonstrate why one project is chosen and another is not. Agency officials told us that given the complex and varied nature of their mandate and organization, they have not been able to develop these evaluation criteria.

5.41 In summary, we found that, in the past two years, the Agency has significantly strengthened its governance framework for managing individual IT projects. It has also taken steps to acquire reliable information about the sustainability of its current suite of IT applications and develop a long-term view of its investments. These initiatives need to be completed. In addition, we found that the Agency could improve its decision making by periodically analyzing its IT investments, according to the various categories in its portfolio, and by clarifying the evaluation criteria that it uses to choose from competing IT investment proposals.

5.42 Recommendation. To strengthen its governance of IT investments and provide management with better information for decision making, the Canada Revenue Agency should

The Agency's response. Agreed. The Canada Revenue Agency (CRA) will complete its inventory of IT applications (including an assessment of their sustainability and associated risk) and will finish developing its strategic investment plan, making every effort to ensure consistency with the Treasury Board Policy on Investment Planning—Assets and Acquired Services. Although the CRA feels that senior management is already well-informed regarding the nature, purpose, and mix of major projects in which the Agency is investing at any given point in time, the introduction of a more formal strategic investment plan will provide an opportunity for the Agency to more clearly define, summarize, and report on the categorization of its current and planned investments.

Many factors must be weighed by senior management when deciding which projects to undertake, and in what order. Each decision is typically made in a unique context, and the criteria used to assess and compare the value and importance of proposed investments are chosen accordingly. The Agency will undertake to better describe that context through categorization of projects that are under way and proposed and to ensure transparency regarding the criteria applied to a particular decision.

A more comprehensive performance-reporting framework is needed

5.43 A comprehensive performance reporting framework is an essential element of managing IT investments in the Agency. We expected to find a performance-reporting framework that would enable the Agency to demonstrate that it manages its IT investments effectively and that it is making investments that will meet the organization's current and future objectives, at a reasonable cost and with an acceptable level of risk.

5.44 We found that the Agency monitors and reports on some aspects of the levels of service that the IT Branch provides to taxpayers. In addition, the Agency periodically reviews its information technology infrastructure, using benchmarks to compare its performance to organizations of similar size and complexity. The IT infrastructure was last reviewed in the 2003–04 fiscal year, and the most recent application review was done in early 2008.

5.45 At the project level, the Agency asked project management teams in early 2008 to start reporting on the realization of the expected benefits at the end of each project. In addition, the Agency reports to its Board of Management about the status of IT projects that exceed $20 million, and it recently started providing the RIMC Secretariat with quarterly updates on the individual IT projects that the RIMC is responsible for.

5.46 Although the Agency has improved its oversight and monitoring of information for individual projects, it provides limited performance information at the portfolio level. For example, it is not providing performance information on the

5.47 Recommendation. The Canada Revenue Agency should develop more comprehensive performance information at the strategic level to help those responsible for managing and overseeing IT investments make more informed decisions.

The Agency's response. Agreed. The enhanced Agency project oversight framework, which was approved by the Agency's Management Committee in early 2008, introduced a number of measures that will help the Agency to develop more comprehensive performance information at the strategic level. These measures include, among other things: the requirement for a more robust performance measurement plan and evaluation strategy for every new project, ensuring that the objectives, expected outcomes, and success criteria are clearly articulated at the outset; a greater focus on risk identification and mitigation as part of the project business case and implementation plan; the introduction of quarterly performance "dashboard" reports for all corporately monitored projects (to supplement the more detailed progress reports that were already required to be submitted periodically); and the introduction of a new Benefits Realization Confirmation step at the end of each project. Taken together, these measures will enable senior managers to acquire a better understanding of the overall health and level of risk in the IT project portfolio, to better assess how well the Agency is achieving its strategic investment objectives, and to better gauge the overall success rate of IT projects.

Managing information technology projects

The Agency has not complied with its own guidance for managing information technology projects

5.48 An organization employs project management techniques to control and coordinate its activities, resources, time, and costs—in short, to ensure that projects deliver the value for which they were approved. Sound project management includes

5.49 To determine whether the Canada Revenue Agency follows sound project management practices in managing IT investments, we examined eight Agency projects that were under development or had recently been completed at the time of our audit (Exhibit 5.2). We assessed whether the selected projects complied with Agency policies, guidelines, and procedures. We also evaluated each project against the following four criteria:

Although there are numerous aspects to project management, we selected the four criteria that, in our opinion, are most closely associated with an organization that wants to demonstrate that it is delivering the value the IT investments were designed to achieve.

Exhibit 5.2—IT projects selected for audit

Project Branch Description Current Budget1 Spent by 31 March 2008 Schedule Status
Compliance Systems Redesign (CSR) Compliance Programs Branch A major business transformation initiative designed to improve the Canada Revenue Agency's (CRA's) capacity to manage and deliver tax compliance programs $97.5 million $40.6 million 2002–11 In development
Network Services Enhancement Project (NSEP) IT Branch—partially funded by the Canada Border Services Agency A network upgrade project $7.9 million $7.1 million 2005–08 Operational
Business Intelligence and Decision Support (BIDS) Assessment and Benefit Services Branch and Information Technology Branch Created to define and implement a corporate approach and facility to deliver Business Intelligence solutions across CRA $50.2 million $40.0 million 2002–09 In development
Working Income Tax Benefit (WITB) Assessment and Benefit Services Branch A refundable tax credit for low income Canadians introduced in the 2007 federal budget—has a prepayment mechanism that begins in 2008 and involved modifications to the T1 assessing system, the benefits system, and related systems $4.5 million $2.7 million 2007–08 Operational
T2 Two-Dimensional Bar Coding Assessment and Benefit Services Branch Used to process computer-generated T2 corporation income tax returns

Bar-coded returns are scanned in the tax centres, and the data is then processed in systems currently used to assess T2 returns—resulting in a more efficient data capture process, with reduced processing times and fewer data entry errors

$2.2 million $1.9 million 2004–06 Operational
Integrated Charities System/ Charities Tracking System Legislative Policy and Regulatory Affairs Branch—a 2004 Treasury Board submission provided the funding for the system An IT-enabled project to help the charities directorate better manage documents and track work-flow—part of an initiative to reform Charities administration at CRA $3.7 million $3.9 million 2004– (ongoing) In development
Taxpayer Relief Registry Redesign Appeals Branch Being redesigned to improve the reporting functions and consistency with which taxpayer relief requests are processed $2.7 million $1.1 million 2005–10 In development
Portageur Service Pilot Assessment and Benefit Services Branch Allows clients to verify their personal identification data with Veterans Affairs and authorizes that Department to share the information through an electronic transfer with the Canada Revenue Agency

The shared authentication allows clients to access their online tax services through a secure and private channel

$0.4 million $0.4 million 2005–06 Operational
1Budget for development costs only—does not include ongoing operations

5.50 Before developing a project management policy in 2006, the Agency used the Enhanced Framework for the Management of Information Technology Projects that was developed by the Treasury Board of Canada Secretariat. Of the eight projects we audited, seven began before the Agency implemented its new project management policy. However, the four criteria we have selected for review were part of the enhanced framework as well as being part of the best practices for IT management that existed at the time. Since they are still relevant and are important elements of the Agency's current policy framework, we expected the Agency to meet our criteria. It should be noted that meeting the criteria does not necessarily guarantee that the projects will be successful and meet their business expectations. However, they would be better managed and have a greater chance of being successful if the criteria are met.

Only two of the eight audited projects met all four criteria

5.51 While all of the projects partially met some of the criteria, only two of the eight projects met all four criteria (Exhibit 5.3).

Exhibit 5.3—Six of eight projects did not meet all criteria

Audit criteria Project CSR Project NSEP Project BIDS Project WITB Project T2 Bar coding Project Charities Project Tax Relief Project Portageur
Comprehensive business case in compliance with best practices Partially met Partially met Not met Met Met Not met Partially met Partially met
Appropriate governance and accountability structures Partially met Partially met Partially met Met Met Not met Partially met Met
Adequate project management based on risk management Met Partially met Partially met Met Met Partially met Met Met
Clearly defined benefits that are adequately tracked and reported Partially met Not met Not met Met Met Not met Not met N/A

Met The criterion attributes were met over the life of the project.

Partially met The criterion attributes were partially met over the life of the project.

Not met The criterion attributes were not met over the life of the project.

N/A - Not assessed

5.52 Business cases. Our first criterion involves making a sound business case, which provides managers with the information they need to decide whether a project should proceed. Making a business case is a critical early activity in the lifecycle of an IT investment.

5.53 Of the eight projects, only two included business cases that met our expectations, while four others partially met our expectations.

5.54 Two projects did not meet our expectations:

5.55 The Business Intelligence and Decision Support (BIDS) project was among those with business cases that did not meet our criterion. To illustrate our concern about the Agency's compliance with guidance related to making businesses cases, we have included the following case study. Some of the information reported in this case study was derived from the work performed by the Corporate Audit and Evaluation Branch and reported in its January 2008 report on the BIDS project.

Case Study 5.1—Business Intelligence Decision Support

The Business Intelligence Decision Support (BIDS) project illustrates how a flawed business case increases the risk that the project will not meet its business expectations.

The BIDS project began in 2002. Its purpose was to implement a corporate approach to consolidating data from widely dispersed national systems within the Canada Revenue Agency to make the data easier to access and use. The project was initially divided into three distinct phases. We found that both the original business case for Phase 1 and the 2006 business case for Phase 2 failed to meet most of our criteria for these documents.

We found that the business cases for phases 1 and 2 excluded significant elements, such as consideration of annual maintenance costs of more than $6 million and the costs associated with analyzing and inputting data into a corporate data warehouse.

We also found that the business cases did not have clearly defined milestones for all three phases and that $46 million of the total project funding ($50.2 million) was not linked to key deliverables.

In addition, the business cases for phases 1 and 2 did not indicate how data quality issues would be addressed. In a project with as many users as BIDS, data quality is a significant issue that needs to be fully understood from the outset.

5.56 Governance and accountability. At the organizational level, governance of information technology requires an appropriate management framework to help ensure that all such technology contributes to achieving an organization's objectives. At the project level, governance focuses on delivering projects that will help organizational units meet their business objectives, at acceptable costs and level of risk. Managers of the Agency's branches and regions are responsible for putting appropriate project management structures in place.

5.57 Overall, only three of the eight projects we audited met our governance criterion. Four other projects partially met our criterion, and one project did not meet our criterion. A common deficiency in these projects was the lack of clearly defined roles and responsibilities for managing the project.

5.58 To illustrate our findings about project governance and accountability, we have included a case study of the Compliance System Redesign project. Specifically, the case study illustrates how the new governance structure has had a positive impact on this large project.

Case Study 5.2—Compliance System Redesign

The Compliance System Redesign project is an example of a project that lacked appropriate governance and oversight for a number of years. This has contributed to significant project delays.

The Compliance System Redesign project was originally part of the Business Integration Systems Support Infrastructure (BISSI). The BISSI was a larger initiative that included a large suite of IT systems designed to help the Compliance Programs Branch (CPB) manage its programs more efficiently and to act as an Agency-wide case management system.

The project was initiated and approved by the Resource Project and Review Committee (RPRC), the predecessor to the Resource and Investment Management Committee (RIMC), in February 2002. The project had initial funding of $3.8 million, a total estimated project cost of between $23 and $31 million, and a forecasted completion date of 2007.

The first full business case for this project was presented to the RIMC in March 2006 after more than $20.4 million was spent. This business case was approved by the Board of Management in December 2006. At that time, the initiative was split up into two projects. In December 2007, the two projects were reunited. Finally, in May 2008, after $40.6 million had been spent, the RIMC recommended a revised business case for approval to the Agency Management Committee. The revised case clearly explains the rationale for the project, the project's design, and how the project would meet the Agency's business needs. It also included a forecasted completion date of 2011 and a total estimated cost of $97.5 million. It should be noted that the reported costs were not all attributable to the development of the business case; the project team did build some of the core functionality required by the project at the same time.

We rated this project as "partially met" for governance and oversight because we found that oversight has been more rigorous since RIMC was created.

5.59 Risk Management. An important component of project management is risk management, which involves

5.60 For an IT project to succeed, its risks must be identified and mitigated. Five of the eight projects we audited met our risk management criteria, and three projects partially met it.

5.61 To illustrate our findings about risk management, we have included a case study of the Integrated Charities System project.

Case Study 5.3—Integrated Charities System

The Integrated Charities System (ICS) is an example of a project that identified the risks at the outset but did not adequately manage them throughout the project's development. For example, the risk that the solution would not be accepted by the end users was identified, but not mitigated. This resulted in the development of an application that did not meet the needs of the Charities Directorate, which it was designed to serve.

In 2004, the Canada Revenue Agency began a project to provide the Directorate with a software application that would improve the way it managed documents and tracked workflow.

We found that this project's business case did not reflect a number of the requirements of CRA policy. For example, the business case did not clearly indicate how the application would contribute to meeting the business or operational needs of the Agency and the Charities Directorate.

While the business case included a number of qualitative benefits, it did not define any quantitative benefits that the system was to produce, nor did it identify the full lifecycle costs associated with the application. In particular, it did not include the cost of ongoing maintenance (which turned out to be substantially more than the IT Branch could afford).

The Directorate told us that officials did not know at the outset what functions the system should have been able to perform to improve business processes. Until the system was almost completed, the Directorate also did not understand what it would look like. These factors all contributed to Agency's inability to deliver this project successfully.

In early 2007, the IT Branch delivered the system to the Charities Directorate. With the Agency reporting that almost $2.6 million had been invested in the system, the Directorate deemed the project as being unable to meet its needs. The Directorate then asked branch personnel to design its own project for the Directorate (called the Charities Tracking System), to better meet its business needs. This process was ongoing at the time of our audit, and the project is not expected to be completed for at least another year.

5.62 Benefits. Specific, quantifiable benefits can serve as milestones or indicators for measuring a project's success and can provide a structured way to track progress. Milestones are points at which an organization can determine whether a project is straying from its original goals and can prompt management to correct any problems.

5.63 We expected that, for each project we reviewed, the Agency would have specified the expected benefits and business outcomes. We note that if an organization does not specify outcomes at a project's outset, it is not possible later to assess that project's performance objectively. One cannot assess whether a project has actually performed against initial expectations if these expectations were never specified.

5.64 Overall, only two of seven projects that we audited met our criteria of specifying expected benefits or outcomes. We could not assess the eighth project for this criterion because it was part of a government-wide initiative that the Agency developed for the benefit of other entities.

5.65 To illustrate compliance with Agency guidance on identifying, measuring, and assessing benefits, we have included a case study of the T2 Two-Dimensional Bar Coding project.

Case Study 5.4—T2 Two-Dimensional Bar Coding Project

The T2 Two-Dimensional Bar Coding project involved placing bar codes on computer-generated T2 corporate income tax returns. It is an example of a project that benefited from a strong planning process that included setting measurable targets and tracking and reporting benefits. The project also built on experience gained from an earlier bar coding project for T1 forms.

At the outset, the project team clearly defined the increased efficiency benefits that the system was expected to deliver and held bi-weekly meetings to discuss the project's status and measure progress against milestones.

The team delivered the project on time, and it met or surpassed the benefits specified during the planning process. The project's benefits included better data quality, because the system eliminated keying errors, as well as more efficient, less expensive, and faster processing.

5.66 The Agency's new project management policy and practices may have prevented some of the problems we identified during our audit. However, we believe project teams need to adhere to the Agency's recommended policies and practices more rigorously to provide senior management and the Board of Management with greater assurance that all future IT projects comply with the Agency's guidelines for business cases, for management of outcomes and benefits, and for appropriate governance and risk management regimes.

Project review procedures need to be strengthened

5.67 Over the last 12 years, other federal governments and the private sector have studied the challenges facing large IT projects and the reasons why so many have not delivered the intended value to the organization. We found that, in response to the high failure rate of surveyed IT projects, several other governments—notably in the United Kingdom and Australia—have adopted new methodologies for monitoring large IT projects as well as a framework for conducting independent third-party gate reviews.

5.68 Our review of eight IT projects indicates that the monitoring of IT projects in the Agency needs to be strengthened. We expected the Agency to have implemented gate reviews or other similar reviews to examine projects at critical stages: initiation, planning, development, and implementation.

5.69 We found that the Agency has implemented certain internal review procedures that reduce the risk of project failure. For example, the RIMC Secretariat reviews all documentation from the project teams and challenges the information provided. Agency officials told us that the documents required for project approval are reviewed extensively by the IT branch. In addition, the RIMC has occasionally asked the Corporate Audit and Evaluation Branch to provide an independent review of the progress of certain projects, generally when they appeared to be experiencing difficulties.

5.70 In our opinion, these procedures are sound but do not replace scheduled gate reviews by independent subject matter experts. In addition to existing procedures, gate reviews are particularly important for high-risk projects, and they need to be conducted by teams of IT subject matter experts, internal auditors, or private independent contractors.

5.71 Recommendation. The Canada Revenue Agency should strengthen its review procedures, including requiring that high-risk IT projects are independently assessed at specific intervals during their lifecycle.

The Agency's response. Agreed. The Canada Revenue Agency will examine ways to further strengthen its review procedures, including the engagement of an independent party to assess high-risk IT projects at critical stages during their lifecycle. The requirement, timing, and scope of such independent reviews will be established at the beginning of the project by the approval authority.

Conclusion

5.72 In our opinion, the Canada Revenue Agency's systems and practices for selecting and managing its information technology investments, as currently designed, provide management with reasonable assurance that it is managing those investments well and that the investments are in line with the Agency's business objectives. However, since some of the key systems and practices are new and IT projects often take many years to complete, we were not able to determine whether the systems and practices were delivering their intended benefits.

5.73 Although we found a number of deficiencies in the management of IT projects that were started before the latest improvements were introduced, the Agency's new systems and practices are designed to reduce the risk that such problems will occur in the future. The Agency will need to monitor these new systems and practices to ensure that they are being implemented as designed and that the intended results are being achieved.

5.74 We believe that the Agency needs to strengthen its governance framework for IT to better support the management of its portfolio of IT investments. We found that the Agency has some portfolio management practices and is considering adopting others. Strengthening portfolio management would allow the Agency to provide better oversight of its information technology investments and would help to ensure that it is selecting projects that best meet its business objectives, with an acceptable degree of risk and at a reasonable cost.

5.75 We found problems in the way six of the eight projects we selected for review were managed. Although clear project management expectations existed in the four areas we examined, they were not all complied with. The deficiencies we found resulted in some significant delays and, in one case, an IT project that was not accepted by the branch or implemented.

5.76 Our findings illustrate that having established systems and practices is not enough. Project teams must respect those systems and practices, and oversight bodies need to regularly monitor whether project teams are following established policies and procedures and whether their progress is on track.

About the Audit

Objective

The objective of our audit was to determine whether the Canada Revenue Agency's systems and practices provide it with reasonable assurance that it is managing its information technology IT investments well, and that they are in line with its business objectives.

Scope and approach

We examined how well the Agency manages its IT investments, using recognized best practices for IT management as well as other best practices the federal government has identified. The scope of the audit included all Agency responsibilities, including its governance and management frameworks. We only reviewed those management practices and procedures within the Agency.

The audit assessed whether the Agency had management practices in place to align its IT investments with its business objectives, and whether those practices demonstrate that the investments are delivered with an acceptable degree of risk and at a reasonable cost. In addition, we examined a selection of IT-enabled business investments to determine whether the Agency was using these management practices effectively. We also verified that the selected projects complied with Agency policies, guidelines, and procedures. We did not assess whether the projects achieved expected outcomes.

We interviewed officials at Headquarters from the Information Technology Branch and in five branches leading the projects that were under way or recently completed. We also interviewed current and previous members of the Resource and Investment Management Committee.

Our approach included analyzing various documents (policies and guidelines) and meeting employees and managers involved in the selected projects. In addition to Headquarters, we visited two Tax Services Offices and one Tax Centre. During those visits, we interviewed senior officials.

In conducting our audit, we relied on two internal audits conducted by the Corporate Audit and Evaluation Branch: Local Solutions and Business Intelligence Decision Support.

Criteria

Listed below are the criteria that were used to conduct this audit and their sources.

Criteria Sources
Making the right investments

We expected that the Agency would have adopted an appropriate governance structure consistent with best practices for IT management.

IT Governance Institute, Enterprise Value: Governance of IT Investments—The Val IT Framework (2006) sections 4 and 5

We expected that the Agency would have established clear direction and strategies for its IT investments, in keeping with its overall corporate objectives.

IT Governance Institute, Enterprise Value: Governance of IT Investments—The Val IT Framework (2006), sections 4 and 5

We expected that the Agency would have implemented appropriate systems and practices to evaluate, prioritize, and choose IT investments.

IT Governance Institute, Enterprise Value: Governance of IT Investments—The Val IT Framework (2006), sections 4 and 5

Managing information technology projects

We expected that the Agency's projects would include

  • comprehensive business cases, in compliance with best practices;
  • appropriate governance and accountability structures;
  • adequate project management, based on risk management; and
  • clearly defined benefits that were adequately tracked and reported.
  • Treasury Board of Canada Secretariat, An Enhanced Framework for the Management of Information Technology Projects—Project Management Guide (February 2002)
  • Canada Revenue Agency, Policies, Procedures, and Guidelines for Managing Projects
  • IT Governance Institute, Enterprise Value: Governance of IT Investments—The Val IT Framework (2006), Section 5

Audit work completed

Audit work for this chapter was substantially completed on 30 May 2008.

Audit team

Assistant Auditor General: John Rossetti
Principals: Richard Brisebois and Jamie Hood
Directors: Greg Boyd, Tony Brigandi, and Martin Ruben

Serge Campeau
Violaine Guillerm
Marie-Claude La Salle
Étienne Robillard

For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free).

Appendix—List of recommendations

The following is a list of recommendations found in Chapter 5. The number in front of the recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses indicate the paragraphs where the topic is discussed.

Recommendation Response
Making the right investments

5.22 Within two to three years, the Agency Management Committee should ensure that it receives and reviews information on how well its new project management policies, procedures, and guidelines are being implemented and on how well they are being complied with throughout the Agency. (5.10–5.21)

Agreed. Within two or three years, the Canada Revenue Agency (CRA) will undertake and complete an assessment of how well the enhanced project approval and monitoring framework is being implemented and complied with across the Agency. This kind of periodic review is performed as a matter of course by the CRA. In fact, such a review led to the above-noted improvements to the project oversight framework, which were approved by the Agency's Management Committee in early 2008. These changes included the development of new training and information products to increase the awareness of the new project management regime, as well as more detailed guidelines and templates to facilitate compliance.

5.42 To strengthen its governance of IT investments and provide management with better information for decision making, the Canada Revenue Agency should

  • complete its inventory of IT applications, including assessing their sustainability and associated risks;
  • finish developing its multi-year strategic investment plan;
  • define appropriate categories for the IT investment portfolio and report the results periodically to senior management; and
  • document clear evaluation criteria for prioritizing and selecting IT investments for the portfolio. (5.23–5.41)

Agreed. The Canada Revenue Agency (CRA) will complete its inventory of IT applications (including an assessment of their sustainability and associated risk) and will finish developing its strategic investment plan, making every effort to ensure consistency with the Treasury Board Policy on Investment Planning—Assets and Acquired Services. Although the CRA feels that senior management is already well-informed regarding the nature, purpose, and mix of major projects in which the Agency is investing at any given point in time, the introduction of a more formal strategic investment plan will provide an opportunity for the Agency to more clearly define, summarize, and report on the categorization of its current and planned investments.

Many factors must be weighed by senior management when deciding which projects to undertake, and in what order. Each decision is typically made in a unique context, and the criteria used to assess and compare the value and importance of proposed investments are chosen accordingly. The Agency will undertake to better describe that context through categorization of projects that are under way and proposed and to ensure transparency regarding the criteria applied to a particular decision.

5.47 The Canada Revenue Agency should develop more comprehensive performance information at the strategic level to help those responsible for managing and overseeing IT investments make more informed decisions. (5.43–5.46)

Agreed. The enhanced Agency project oversight framework, which was approved by the Agency's Management Committee in early 2008, introduced a number of measures that will help the Agency to develop more comprehensive performance information at the strategic level. These measures include, among other things: the requirement for a more robust performance measurement plan and evaluation strategy for every new project, ensuring that the objectives, expected outcomes, and success criteria are clearly articulated at the outset; a greater focus on risk identification and mitigation as part of the project business case and implementation plan; the introduction of quarterly performance "dashboard" reports for all corporately monitored projects (to supplement the more detailed progress reports that were already required to be submitted periodically); and the introduction of a new Benefits Realization Confirmation step at the end of each project. Taken together, these measures will enable senior managers to acquire a better understanding of the overall health and level of risk in the IT project portfolio, to better assess how well the Agency is achieving its strategic investment objectives, and to better gauge the overall success rate of IT projects.

Managing information technology projects

5.71 The Canada Revenue Agency should strengthen its review procedures, including requiring that high-risk IT projects are independently assessed at specific intervals during their lifecycle. (5.48–5.70)

Agreed. The Canada Revenue Agency will examine ways to further strengthen its review procedures, including the engagement of an independent party to assess high-risk IT projects at critical stages during their lifecycle. The requirement, timing, and scope of such independent reviews will be established at the beginning of the project by the approval authority.

 


Definitions:

Information technology investments—In this chapter, existing business applications, networks and hardware, and projects under way to update, replace, or enhance the existing IT infrastructure. (Return)

Gate reviews—Projects are reviewed at key decision points by a team of experienced people who are independent of the project team. The purpose of these reviews is early detection of problems that could threaten the success of projects. (Return)

 

PDF Versions

To access the Portable Document Format (PDF) version you must have a PDF reader installed. If you do not already have such a reader, there are numerous PDF readers available for free download or for purchase on the Internet: