Report on a Review of the Direct Engagement Audit Practice—Direct Engagement Audits Completed in the 2016–17 Fiscal Year
Report on a Review of the Direct Engagement Audit Practice—Direct Engagement Audits Completed in the 2016–17 Fiscal Year
Table of Contents
- Results of the Reviews
- Appendix A—System of Quality Control Elements
- Appendix B—System of Quality Control Elements and Process Controls Reviewed
1. The Office of the Auditor General of Canada (the Office) conducts independent audits and studies that provide objective information, advice, and assurance to Parliament, territorial legislatures, boards of Crown corporations, government, and Canadians. The Office carries out three main types of legislative audits: financial audits, performance audits, and special examinations. Performance audits and special examinations are referred to as direct engagements.
2. A performance audit is an independent, objective, and systematic assessment of how well government is managing its activities, responsibilities, and resources. Performance audits contribute to the effectiveness of the public service and the accountability of the government to Parliament and Canadians. Performance audits are planned, performed, and reported in accordance with professional auditing standards and Office policies.
3. Special examinations are a form of performance audit that is conducted within Crown corporations. The Office audits most, but not all, Crown corporations. The scope of special examinations is set out in the Financial Administration Act. A special examination considers whether a Crown corporation’s systems and practices provide reasonable assurance that its assets are safeguarded, its resources are managed economically and efficiently, and its operations are carried out effectively.
4. The mission of the Practice Review and Internal Audit team is to enhance and protect organizational value by providing risk-based and objective assurance, advice, and insight. The team helps the Office accomplish its objectives by offering management recommendations based on the application of a systematic, disciplined approach to evaluating and approving the design and effectiveness of risk management, control, and governance processes.
5. The team helps the Office meet its obligations under Canadian Standard of Quality Control 1 of the Chartered Professional Accountants of Canada. It does this by conducting inspections to determine the extent to which engagement leaders are complying with professional standards, Office policies, and applicable legislative and regulatory requirements when conducting their audits. These inspections also help to ensure that audit reports are supported and appropriate.
6. The team also performs its work in accordance with the Office’s most recent Practice Review and Internal Audit Plan, as recommended by the Audit Committee and approved by the Auditor General. The Plan is based on systematic, cyclical monitoring of the work of all engagement leaders in the Office.
7. To ensure that audits meet the standards of the Chartered Professional Accountants of Canada, the Office establishes policies and procedures for its work. These are outlined in the Office’s Direct Engagement Manual, in its System of Quality Control, and in various other audit tools that guide auditors through the required steps. The three assistant auditors general responsible for direct engagement audits provide leadership and oversight of the Office’s direct engagement audit practice and contribute to the quality of individual audits.
8. The Office’s direct engagement methodology update in the fall of 2015 included significant changes to the Direct Engagement Manual, TeamMate audit procedures, and templates. These changes were made largely to ensure compliance with the new Canadian Standard for Assurance Engagements (CSAE) 3001—Direct Engagements. These changes were to be applied gradually, depending upon when a specific direct engagement would be reporting, up to the spring of 2017. The Practice Review and Internal Audit team’s methodology is consistent with these changes.
9. This report summarizes the key observations related to the practice reviews of selected direct engagement audits completed in the 2016–17 fiscal year.
10. The objective of practice review is to provide the Auditor General with assurance that
- direct engagement audits comply with professional standards, Office policies, and applicable legislative and regulatory requirements; and
- audit reports are supported and appropriate.
Scope and methodology
11. The Practice Review and Internal Audit team conducted practice reviews of six direct engagement audits (five performance audits and one special examination) that had been completed in the 2016–17 fiscal year. Our methodology requires that we review a selection of completed audits on a cyclical basis, including at least one engagement for each engagement leader over a four-year monitoring cycle. We used a random sampling approach to select the engagement leaders and their related files.
12. Our reviews included an examination of electronic (TeamMate) files as well as paper files, if applicable. We reviewed documentation related to the planning, examination, and reporting of the audits. We also interviewed quality reviewers, and we met selected audit team members and other internal specialists, as needed, to discuss issues.
13. We reviewed all files selected in terms of the System of Quality Control (Appendix A). We focused our work on the selected elements and process controls that we considered to be key or high risk (Appendix B) in the selected audits.
14. For each audit we reviewed, we rated each selected System of Quality Control element and process control as one of the following:
- Compliant. Performance is satisfactory, with minor improvement possible; the audit file is in compliance, in all significant respects, with General Assurance and Auditing Standards (GAAS) and Office policies.
- Compliant while improvements needed. Improvements are necessary in one or more areas to fully comply with GAAS and Office policies.
- Non-compliant. Significant deficiencies exist; the audit does not comply with GAAS or Office policies.
15. After completing each practice review, we concluded whether the independent audit opinion was supported and appropriate. We also concluded whether the audit file was compliant overall with GAAS and with Office policies.
Results of the Reviews
Appropriateness of the audit reports
16. Overall, we found that the audit reports were supported and appropriate in the six files we reviewed.
Compliance with the System of Quality Control elements and process controls
17. In general, the overall level of compliance with the System of Quality Control elements was good. All six files were compliant while improvements were needed. For more information, see the Observations section.
18. It is important to note that our overall conclusion on a specific file is based on the review of all elements of the System of Quality Control. Consequently, it is possible to be non-compliant with one element of the System of Quality Control even though the overall conclusion is “compliant while improvements needed.”
19. For the current practice review cycle for both financial audits and direct engagements, we have performed a more detailed review of the Independence Confirmation forms compared with the previous review cycle.
20. The Office has established policies and procedures for independence, which are documented in both the financial audit and direct engagement practice manuals. Both manuals outline the following policy in Section 3031—Independence:
All individuals who meet the definition of an engagement team member, including internal and, where appropriate, external specialists, shall confirm their independence before commencing work on the engagement. [NovemberNov-2011]
21. Our understanding is that this policy requirement is intended to ensure that all threats to independence are identified on a timely basis so that their significance can be assessed, and so that safeguards can be put in place to reduce or eliminate all significant threats to an acceptable level.
22. We found that the six files we reviewed did not comply with one or more of the requirements of the Office’s policy on independence. It is important to note that no threats to independence were identified in the files that we reviewed.
23. We found that four of the files we reviewed were missing Independence Confirmation forms for individuals who met the definition of an engagement team member. In total, 10 Independence Confirmation forms were missing from these files. In these cases, we have asked the engagement leaders to reopen the audit files to ensure that independence is assessed and documented for each team member. We have asked them to inform the Chief Audit Executive if any conflicts are identified.
24. We also found that some engagement team members had charged time to engagements before preparing their Independence Confirmation forms. We reviewed 94 completed forms and found that more than one third of engagement team members charged time to engagements before completing their forms. On average, these individuals charged about 15 hours to the engagements before completing the forms. We identified some cases in which the individuals had charged more than 30 hours to the engagements over a period of many months before completing the forms.
25. We believe that this is a systemic matter requiring one or more of the following: corrective action, changes to the Office’s policy, or changes to the Office’s procedures.
26. The Office’s Direct Engagement Manual also states the following in Section 3031—Independence:
The engagement leader shall form a conclusion on team members’ compliance with independence requirements that apply to the assurance engagement. [Nov-2011]
27. To help employees interpret some of its policies, the Office has developed a document entitled Independence—Frequently Asked Questions (FAQ). Question 15 of this document is “What should be done with a completed Independence Confirmation?” The response states that a “completed Independence Confirmation must be reviewed by the engagement leader before the engagement team member commences work on the assurance engagement.”
28. We consulted with the Office’s Internal Specialist—Values and Ethics and representatives from the Direct Engagement Practice Team to obtain their views on question 15 of the FAQ document. We were informed that, although the policy does not require leaders to review the Independence Confirmation forms before team members commence work, the question had been developed to minimize the risk that engagement leaders would delay their review and approval of the forms. The objective is to ensure that each engagement leader has taken appropriate action against potential threats to independence reported in the Independence Confirmation forms. This must be done on a timely basis.
29. We found delays in the engagement leader’s review and approval of the Independence Confirmation forms in five of the six audit files we reviewed. We rated these five files as compliant while improvement was needed. A total of 80 Independence Confirmation forms had been prepared for these five files. We noted that in about half of the forms we reviewed, individuals had charged an average of 40 hours to the audit before the engagement leader had reviewed and approved the form. We found seven cases in which more than 100 hours had been charged to the audit before the Independence Confirmation form was reviewed and approved.
30. We believe that this matter is also a systemic one that requires one or more of the following: corrective action, changes to the Office’s policy, or changes to the Office’s procedures.
31. Recommendation to the Direct Engagement Audit Practice. Engagement leaders should
- ensure that engagement team members confirm their independence before commencing work on an engagement, and
- confirm the independence of engagement team members by reviewing and approving each member’s Independence Confirmation form before the member begins working on an engagement.
Management’s response. Agreed. Practice engagement leaders should comply with policy and confirm independence of engagement team members to ensure that there are no significant threats to independence. The Office is in the process of updating its policy to clarify the timing of such confirmations, to ensure that policy requirements be timely yet practical.
32. Recommendation to Audit Services. Audit Services should assess whether changes are required to the independence confirmation process or policy, or both.
Management’s response. Agreed. Audit Services had previously identified opportunities to improve the efficiency and effectiveness of the independence confirmation process and has submitted a project proposal to Information Technology Services to improve the process through greater use of automation and the Office’s time reporting system. In the interim, while awaiting project resources, Audit Services will update the audit methodology concerning independence to improve its design and operating effectiveness.
Security of sensitive information
33. In our Report on a Review of the Direct Engagement Audit Practice—Direct Engagement Audits Completed in the 2015–16 Fiscal Year, we noted that audit staff needed to be made aware of the Office’s security policy, and that any document stored in TeamMate should be assessed against the policy and be labelled according to the proper security level. In our review of this year’s files, we note that work still remains to be done in applying the Office’s security policy. Five of the six files we reviewed included documents that were not properly labelled in accordance with the Office’s security policy.
Supervision and review
34. We noted that in the area of supervision and review, all of the six files complied, though four needed improvements.
35. In two files, we noted that the audit team had not used some of the most recent templates available at the time of the audit, thereby risking non-compliance with the current Office methodology. In the first file, the audit team had not updated its TeamMate file after a methodology update. As a result, out-of-date templates were used to prepare key documents, and some TeamMate sign-offs were missed. In the second file, the team had used an outdated template to create its Audit Logic Matrix (ALM). As a result, the ALM did not include consideration of some required elements, such as audit risk. Engagement leaders must ensure that required changes in the methodology are properly documented in the audit files; this includes access to the most up-to-date templates, to ensure that audits are carried out in compliance with Office policies.
36. In two files, we did not see evidence that the engagement leaders had reviewed documentation (some key documents in one file; certain audit evidence in the other) supporting high-risk findings, before sending the principal’s draft to the entities. Engagement leaders must be satisfied that there is sufficient appropriate audit evidence in the audit file to support factual statements.
37. In one of the files we reviewed, the TeamMate file had been reviewed and closed without including a written acknowledgement letter from the deputy head confirming the entity’s responsibility for the subject matter, acceptance of the terms of the engagement, and the suitability of the audit criteria. The letter was subsequently located after the practice review had begun. We assessed this as compliant while improvements were needed. Engagement leaders must ensure that the team has obtained acknowledgement from the entity and included this information in the audit file.
Evidence-gathering methods—reliance on secondary evidence
38. During our review of the substantiation of high-risk areas, we noted that, to support an audit finding that included a “we found” statement, one of the six teams relied upon a conclusion of an evaluation completed by the audit entity. The team did not conduct any additional work to verify the relevance, reliability, and validity of the evaluation’s conclusion.
Quality control review
39. A quality reviewer had been assigned to three of the files selected for our review. In two files, the work performed by the quality reviewers complied with Office policy requirements. In a third file, we found that the engagement quality control review was non-compliant. We could not confirm that all minimum quality reviewer responsibilities had been met. For example, we found no evidence in the audit file that the quality reviewer had reviewed key audit documents, including independence and exceptions reports and the engagement risk assessment. As a result, the assurance report was dated and issued despite an incomplete and not fully documented quality review.
40. We found during our review of one file that the team did not meet two key T-minus dates. The transmission draft had been sent to the entities very late; accordingly, publication approval was granted late. The team had not reported any issues with meeting key T-minus dates through the early part of the reporting phase. However, within a month, the team had reported that it was at high risk of not meeting its timelines. We could not find any documentation in the file establishing a new plan to ensure that the audit could be completed within the time frame. Moreover, we could find no evidence in the file that the team had communicated this delay to the entities.
41. In our review of one file, we found that the final audit report did not contain a clear conclusion against the overall audit objective.
Date of the report
42. Our review of one file found that the audit report was dated before the engagement leader had reviewed the audit documentation and before written representations had been obtained from the entity’s management.
43. For all of the direct engagement audit files we reviewed, we concluded that the audit reports were supported and appropriate.
44. We concluded that all six files were compliant while improvements were needed.
Appendix A—System of Quality Control Elements
This diagram shows three sides of a cube, each side depicting aspects of the System of Quality Control.
The top of the cube shows the objectives of the System of Quality Control:
- Compliance with professional standards and applicable legal and regulatory requirements; and
- Reports issued are appropriate in the circumstances.
The right side of the cube shows the two levels of the System Quality Control:
- Firm level (Canadian Standards for Quality ControlCSQC 1)
- Engagement level (Canadian Auditing StandardCAS 220 or Canadian Standard for Assurance EngagementsCSAE 3001)
The left side of the cube shows the elements of the System of Quality Control:
- ethics and independence,
- acceptance and continuance,
- human resources,
- engagement performance, and
Appendix B—System of Quality Control Elements and Process Controls Reviewed
Our review covers the following System of Quality Control elements:
- ethics and independence,
- acceptance and continuance,
- human resources, and
- engagement performance.
Leadership. We reviewed whether the engagement leaders ensured that the audits were carried out in compliance with Office policies, professional standards, the System of Quality Control, and applicable legal and regulatory requirements.
Ethics and independence. We reviewed whether the engagement leaders ensured that the independence of all individuals performing audit work, including specialists, had been properly assessed and documented.
Acceptance and continuance. For initial or recurring engagements, we reviewed whether engagement leaders assessed that the team had the necessary competence, capability, time, and resources; that the team complied with relevant ethical requirements; and that it considered management’s integrity.
Human resources. We reviewed whether the engagement leaders assessed the audit team’s adequacy, availability, proficiency, competence, and resources, and whether they documented their assessments.
Within the engagement performance element, we also assessed the following:
- Supervision and review. We reviewed whether engagement leaders ensured that the audit files had documentation regarding who reviewed the audit work performed, the date, and the extent of the review.
- Consultation. We reviewed whether the engagement leaders ensured that appropriate consultations took place in a timely manner, when required.
- Engagement quality control review. We reviewed whether the quality reviews were carried out in a timely manner and whether the quality reviewers performed objective evaluations of the significant judgments made by the teams, the conclusions reached in supporting the auditor’s reports, and other significant matters.
- Differences of opinion. If differences of opinion occurred, we reviewed whether the engagement leaders followed the Office’s established processes for addressing them.
- Engagement documentation. We reviewed whether engagement leaders properly addressed the confidentiality, safe custody, integrity, accessibility, retrievability, and retention of documentation, and whether the final assembly of the engagement files were completed on a timely basis (that is, the 60-day rule).
Other General Assurance and Auditing Standards requirements and Office policies
We reviewed whether engagement leaders ensured that the audit was planned, executed, and reported in accordance with General Assurance and Auditing Standards, applicable legislation, and Office policies and procedures.
We also considered whether the Office met its reporting responsibilities by having in place appropriate audit methodology, recommended procedures, and practice aids to support efficient audit approaches and to produce sufficient audit evidence at the appropriate time.