Report on a Review of the Direct Engagement Audit Practice—Direct Engagement Audits Completed in the 2017–18 Fiscal Year
Report on a Review of the Direct Engagement Audit Practice—Direct Engagement Audits Completed in the 2017–18 Fiscal Year
Table of Contents
- Results of the Reviews
- Appendix A—System of Quality Control Elements
- Appendix B—System of Quality Control Elements and Process Controls Reviewed
1. The Office of the Auditor General of Canada conducts independent audits and studies that provide objective information, advice, and assurance to Parliament, territorial legislatures, boards of Crown corporations, government, and Canadians. The Office carries out three main types of legislative audits: financial audits, performance audits, and special examinations. Performance audits and special examinations are referred to as direct engagements.
2. A performance audit is an independent, objective, and systematic assessment of how well government is managing its activities, responsibilities, and resources. Performance audits contribute to the effectiveness of the public service and the accountability of the government to Parliament and Canadians. Performance audits are planned, performed, and reported in accordance with professional auditing standards and Office policies.
3. Special examinations are a form of performance audit that is conducted within Crown corporations. The Office audits most, but not all, Crown corporations. The scope of special examinations is set out in the Financial Administration Act. A special examination considers whether a Crown corporation’s systems and practices provide reasonable assurance that its assets are safeguarded, its resources are managed economically and efficiently, and its operations are carried out effectively.
4. The mission of the Practice Review and Internal Audit team is to enhance and protect organizational value by providing risk-based and objective assurance, advice, and insight. The team helps the Office accomplish its objectives by offering management recommendations based on the application of a systematic, disciplined approach to evaluating and approving the design and effectiveness of risk management, control, and governance processes.
5. The team helps the Office meet its obligations under Canadian Standard on Quality Control 1, which is issued by the Auditing and Assurance Standards Board. The team does this by conducting inspections to determine whether engagement leaders (audit leaders) are complying with professional standards, Office policies, and applicable laws and regulations when conducting their audits. It also ensures that audit reports are supported and appropriate.
6. The team performs its work in accordance with the Office’s most recent Practice Review and Internal Audit Plan, as recommended by the Audit Committee and approved by the Interim Auditor General. The Plan is based on systematic, cyclical monitoring of the work of all engagement leaders in the Office.
7. To ensure that audits meet the standards of the Chartered Professional Accountants of Canada, the Office establishes policies and procedures for its work. These are outlined in the Office’s Direct Engagement Manual, in its System of Quality Control, and in various other audit tools that guide auditors. The four assistant auditors general responsible for direct engagement audits provide leadership and oversight of the Office’s direct engagements and contribute to the quality of individual audits.
8. This report summarizes the key observations related to the practice reviews of selected direct engagement audits completed in the 2017–18 fiscal year.
9. The objective of practice review is to provide the Interim Auditor General with assurance that
- direct engagement audits comply with professional standards, Office policies, and applicable laws and regulations; and
- audit reports are supported and appropriate.
Scope and methodology
10. The Practice Review and Internal Audit team conducted practice reviews of seven direct engagement audits (three performance audits and four special examinations) completed in the 2017–18 fiscal year. Our methodology requires that we review a selection of completed audits on a cyclical basis, including at least one engagement for each engagement leader over a four-year monitoring cycle. We used a random sampling approach to select the engagement leaders and their related files.
11. Our reviews included examinations of electronic (TeamMate) files as well as paper files, if applicable. We reviewed documentation related to the planning, examination, and reporting of the audits. We also met with selected audit team members and other internal specialists to discuss issues.
12. We reviewed all files selected in terms of the System of Quality Control (Appendix A). We focused our work on the selected elements and process controls that we considered key or high-risk (Appendix B) in the selected audits.
13. For each audit reviewed, we rated each selected System of Quality Control element and process control as one of the following:
- Compliant. Performance is satisfactory, with minor improvement possible. The audit file is in compliance, in all significant respects, with General Assurance and Auditing Standards (GAAS) and Office policies.
- Compliant while improvements needed. Improvements are necessary in one or more areas to fully comply with GAAS and Office policies.
- Non-compliant. Significant deficiencies exist; the audit does not comply with GAAS or Office policies.
14. After completing each practice review, we concluded on whether the independent audit opinion was supported and appropriate. We also concluded on whether the audit file was compliant overall with GAAS and with Office policies.
Results of the Reviews
Appropriateness of the audit reports
15. Overall, we found that the audit reports were supported and appropriate in all seven files reviewed.
Compliance with the System of Quality Control elements and process controls
16. In general, the overall level of compliance with the System of Quality Control elements was good. All seven files were compliant while improvements needed. For more information, see the Observations section.
17. Our overall conclusion on a file is based on our review of all elements of the System of Quality Control. Consequently, a file can be non-compliant with one element of this system even though the overall conclusion is compliant while improvements needed.
Special consideration for this year’s cycle
18. When performing its reviews for this year’s cycle, the Practice Review and Internal Audit team paid special attention to potential efficiency gains when audit teams performed their work and to good practices that may benefit other teams. The team also looked at ways to gain efficiencies in its own processes. For example, we looked at ways to be more efficient in this year’s review of the independence process.
19. In last year’s summary report, the team recommended that Audit Services assess whether changes were required to the independence confirmation process, policy, or both. Audit Services has since made this assessment and decided to modify the Office’s policy to remove the obligation for team members to confirm their independence before they start working on an audit.
20. The files we reviewed in the current cycle were completed before changes to the Office’s Policy on Independence (in Section 3031 of the Direct Engagement Manual) were considered, approved, and in effect. We expect that an assessment of these files against the standard in effect at the time of the audit would yield observations similar to those identified in the last review cycle. Therefore, we assessed these files against the new policy.
21. The Office has established policies and procedures for independence, which are documented in the Direct Engagement Manual. The manual outlines the following policy in Section 3031 (Independence):
All individuals who meet the definition of an engagement team member, including internal and, where appropriate, external specialists, shall confirm their independence before commencing work on the engagement. [NovemberNov-2011]
22. We found six files that were non-compliant with the requirements of the Office’s Policy on Independence. In these six files, a total of seven Independence Confirmation forms for individuals who met the definition of an engagement team member were missing. It is important to note that no independence issues were identified in the files that we reviewed.
23. We believe that this matter is a systemic issue that requires corrective action.
24. Recommendation on direct engagements. Engagement leaders should be reminded to ensure that engagement team members confirm their independence once they meet the definition of a team member.
Management’s response. We agree with the recommendation. We will remind engagement leaders of their responsibility to ensure that engagement team members confirm their independence once they meet the definition of team members. We will also continue ongoing discussions with Audit Services to explore alternative practices to implement the intent of the standard in more efficient ways.
Assembly of the audit file
25. We found three files that were non-compliant with the requirements of the Office’s Policy on Assembly of the Audit File. The three files were not assembled and finalized within 60 calendar days. Instead, they were assembled and finalized 9, 10, and 25 days after the prescribed time frames.
26. We found three files that were compliant while improvements needed with the requirements of the Office’s Policy on Consultations. In the first file, we noted that the audit team identified the need to consult with an internal specialist at the planning phase; however, we found no documentation of that consultation in the file. In the two other files, consultations were initiated with internal specialists during the planning phase, but we found no documentation related to the agreement by both the individual seeking consultation and the party consulted.
Management of controlled documents
27. We found one file that was compliant while improvements needed with the requirements of the Office’s Policy on Management of Controlled Documents. In that file, the Tracking Sheet for Controlled Paper Copies was incomplete, indicating that more than 17 controlled paper copies had not been returned by the entity. The TeamMate file had been reviewed and closed without documenting the return of the controlled paper copies to the Office. However, after a discussion with the engagement leader, the practice review team was provided with an updated Tracking Sheet for Controlled Paper Copies (kept outside TeamMate) that indicated that all copies had in fact been returned by the entity.
28. We found one file that was non-compliant and one file that was compliant while improvements needed with the requirements of the Office’s Policy on Substantiation (review). In the first file, the engagement leader did not document the identification or review of high-risk paragraphs related to issuing the principal’s (PX) draft. In the second file, there was insufficient evidence that the engagement leader had adequately reviewed the supporting working papers and documents supporting the high-risk paragraphs.
29. We found one file that was compliant while improvements needed with the requirements of the Office’s Policy on Substantiation (documentation). In that file, there were, in our view, some elements of insufficient appropriate audit evidence and a lack of precise references related to the report’s substantiation.
30. We found one file that was non-compliant with the requirements of the Office’s Policy on Third-Party Reference. In that file, the audit team did not verify the accuracy and completeness of statements in audit reports that refer to third parties.
Report content approval and date of the report
31. We found one file that was non-compliant with the requirements of the Office’s Policy on Report Content Approval and Date of the Report. In that file, the audit report was dated before the engagement leader had confirmed its review of the audit documentation and obtained the internal specialist’s sign-off.
Retention policies and procedures
32. We found one file that was non-compliant with the requirements of the Office’s Policy on Retention Policies and Procedures. In that file, the Audit File Completion section was not properly documented, making it impossible to know if any audit records outside of the electronic audit file had been kept.
Office’s guidance for substantiation
33. In conducting our practice reviews, we found that the guidance surrounding substantiation required some clarifications. The policy in Section 1111 (Nature, purpose, and extent of the audit documentation) of the Direct Engagement Manual states, “Auditors shall prepare audit documentation adhering to the experienced auditor principle… .” The guidance in Section 7060 (Substantiation) includes a reference to the concept of the re-performance principle and specifies, “The substantiation prepared for the final report should be prepared in accordance with the ‘re-performance principle,’ meaning another experienced auditor, unfamiliar with the subject matter could review the evidence and accompanying explanation, and be persuaded to reach the same conclusion.” This same guidance requires the substantiation to include precise references to the evidence that supports the findings.
34. On the basis of this policy and guidance, the practice review team expected that substantiation would be performed with adherence to the experienced auditor and re-performance principles. However, the Canadian Standard for Assurance Engagements (CSAE) 3001—Direct Engagements does not refer to the re-performance principle. Rather, it refers only to the experienced auditor principle. As such, the methodology may be confusing the terms “experienced auditor principle” and “re-performance principle.” Furthermore, documentation, evidence, and substantiation are different concepts that are sometimes used interchangeably in the audit manual’s substantiation section. We believe that on the basis of our work, the Office’s guidance for substantiation could be clarified.
35. Recommendation to Audit Services. Audit Services should review the guidance in Section 7060 (Substantiation) of the Direct Engagement Manual for clarity and revise as required.
Audit Service’s response. Agreed. Audit Services has reviewed and clarified Manual section 7060 in its July 2019 methodology update.
Training for special examination auditors
36. More special examinations than usual were performed in this fiscal year as the Office reached the peak of the 10-year cycle. As a result, we have reviewed a larger number of special examination files. There were a number of observations related to file documentation, particularly with regard to the review of file substantiation. We believe that this is linked, to some extent, to the fact that financial auditors are performing special examinations but are not familiar enough with the direct engagement methodology. The special examination engagement leaders may consider providing on-time training, as needed, to the attest auditors who plan to perform special examinations.
37. All seven of the direct engagement files that we reviewed complied with professional standards, Office policies, and applicable laws and regulations, but improvements were needed.
38. Audit reports for all seven of the direct engagement audit files that we reviewed were supported and appropriate.
Appendix A—System of Quality Control Elements
This diagram shows three sides of a cube, each side depicting aspects of the System of Quality Control.
The top of the cube shows the objectives of the System of Quality Control:
- Compliance with professional standards and applicable legal and regulatory requirements; and
- Reports issued are appropriate in the circumstances.
The right side of the cube shows the two levels of the System Quality Control:
- Firm level (Canadian Standards for Quality ControlCSQC 1)
- Engagement level (Canadian Auditing StandardCAS 220 or Canadian Standard for Assurance EngagementsCSAE 3001)
The left side of the cube shows the elements of the System of Quality Control:
- ethics and independence,
- acceptance and continuance,
- human resources,
- engagement performance, and
Appendix B—System of Quality Control Elements and Process Controls Reviewed
Our review covered the following System of Quality Control elements:
- ethics and independence,
- acceptance and continuance,
- human resources, and
- engagement performance.
Leadership. We reviewed whether the engagement leaders ensured that the audits were carried out in compliance with Office policies, professional standards, the System of Quality Control, and applicable laws and regulations requirements.
Ethics and independence. We reviewed whether the engagement leaders ensured that the independence of all individuals performing audit work, including specialists, had been properly assessed and documented.
Acceptance and continuance. For initial or recurring engagements, we reviewed whether engagement leaders assessed that the team had the necessary competence, capability, time, and resources; that the team complied with relevant ethical requirements; and that it considered management’s integrity.
Human resources. We reviewed whether the engagement leaders assessed the audit team’s adequacy, availability, proficiency, competence, and resources, and whether they documented their assessments.
Engagement performance. Within the engagement performance element, we also assessed the following:
- Supervision and review. We reviewed whether engagement leaders ensured that the audit files had documentation regarding who reviewed the audit work performed, the date, and the extent of the review.
- Consultation. We reviewed whether the engagement leaders ensured that appropriate consultations took place in a timely manner when required.
- Engagement quality control review. We reviewed whether the quality reviews were carried out in a timely manner and whether the quality reviewers performed objective evaluations of the significant judgments made by the teams, the conclusions reached in supporting the auditor’s reports, and other significant matters.
- Differences of opinion. If differences of opinion occurred, we reviewed whether the engagement leaders followed the Office’s established processes for addressing them.
- Engagement documentation. We reviewed whether engagement leaders properly addressed the confidentiality, safe custody, integrity, accessibility, retrievability, and retention of documentation, and whether the final assembly of the engagement files were completed on a timely basis (that is, the 60-day rule).
Other Canadian Auditing Standards requirements and Office policies
We reviewed whether engagement leaders ensured that the audit was planned, executed, and reported in accordance with Canadian Auditing Standards, applicable legislation, and Office policies and procedures.
We also considered whether the Office met its reporting responsibilities by having in place appropriate audit methodology, recommended procedures, and practice aids to support efficient audit approaches and to produce sufficient audit evidence at the appropriate times.