This Web page has been archived on the Web.

2008 December Report of the Auditor General of Canada Chapter 8—Reporting on Health Indicators—Health Canada

2008 December Report of the Auditor General of Canada

Chapter 8—Reporting on Health Indicators—Health Canada

Main Points

Introduction

Agreements on health care system renewal
Comparable health indicators
Healthy Canadians—Health Canada's response to the First Ministers' comparable health indicator reporting commitments
Federal responsibility for health care services
Roles and responsibilities for health information and reporting
Recent developments related to the health agreements
Focus of the audit

Observations and Recommendations

Reporting on health indicators

Health Canada has met the specific health indicator reporting obligations

Improvement over time

Healthy Canadians lacks interpretation of progress on health care renewal
Health Canada does not know if recent editions of Healthy Canadians meet the needs of Canadians
Healthy Canadians has not improved over time
Reporting on federal populations is inconsistent across editions of Healthy Canadians
Limited First Nations and Inuit health indicator information is reported in Healthy Canadians
Healthy Canadians receives little media attention
Health Canada's position on the Health Council of Canada's recommendations is unclear
Health Canada needs to devise a strategy to improve Healthy Canadians

Conclusion

About the Audit

Appendix—List of recommendations

Exhibits:

8.1—Benefits of measuring, tracking, and reporting on performance

8.2—Health indicators reported in Healthy Canadians 2006

8.3—Healthy Canadians 2006 presentation of self-reported patient satisfaction with physician care indicator

8.4—Healthy Canadians cover federal populations inconsistently

8.5—Disparity in health status between First Nations and Inuit and the Canadian population

Main Points

What we examined

In 2000, the Government of Canada and provincial and territorial governments reached an historic agreement on health that set out a vision, principles, and an action plan for health system renewal. The First Ministers' commitments on health also called for improvements in accountability and reporting to Canadians and directed federal, provincial, and territorial health ministers to develop indicators that could be compared across jurisdictions and over time to measure progress on renewal. All jurisdictions later committed to public reporting every two years on a number of health indicators—for example, wait times and patient satisfaction with health services. First Ministers' agreements in 2003 and 2004 further reiterated reporting requirements.

Every two years, Health Canada produces the federal report on comparable health indicators, Healthy Canadians: A Federal Report on Comparable Health Indicators, as its response to the federal commitments to health indicator reporting made in the agreements. The report provides selected information on the general Canadian population and on population groups for which the federal government provides health services, such as military personnel and First Nations and Inuit populations. We examined to what extent the Department's reporting on health indicators met the commitments made in the First Ministers' health agreements. We also looked at whether its reporting has improved over time.

Why it's important

The three federal-provincial-territorial agreements represented an attempt by governments to promote renewal of the health care system. The agreements were accompanied by significant increases in federal transfers of funds to provinces and territories. The First Ministers also committed to improve public reporting to Canadians on the progress of health care renewal. The health indicators reports are an important vehicle for enhancing transparency and accountability. Public reporting by governments promotes accountability in a number of ways—for example, by allowing Canadians to see the extent to which governments are attaining their objectives and goals and assisting individuals, governments, and health care providers to make more informed choices.

What we found

The Department has responded. The Department agrees with our recommendations. Its detailed responses follow each recommendation throughout the Chapter.

Introduction

Agreements on health care system renewal

8.1 In the late 1990s, the Canadian health care system was generally seen as being in crisis. Public opinion surveys revealed that Canadians were very concerned about the quality and sustainability of the health care system. They felt strongly that Canadian governments needed to do more to fix the system.

8.2 In light of this crisis, the Government of Canada and provincial and territorial governments decided, through a series of agreements, to strengthen and renew Canada's publicly funded health care system. These agreements also sought to improve accountability and reporting to Canadians. The 2000 First Ministers' Health Communiqué (the 2000 Health Communiqué) articulated a vision, principles, and an action plan for health care system renewal that was accompanied by an increase in health care funding of $21.2 billion over five years. This historic agreement underlined the federal, provincial, and territorial governments' commitment to revitalizing Canada's health care system. Two other agreements were accompanied by further funding—the 2003 First Ministers' Accord on Health Care Renewal (the 2003 Health Accord), which was supported by $36.8 billion over five years, and the 2004 First Ministers' 10-Year Plan to Strengthen Health Care (the 2004 10-Year Plan), which was supported by $41.3 billion over 10 years.

8.3 Health care renewal refers to the federal, provincial, and territorial governments' efforts to strengthen and improve the health care system and ultimately the health of Canadians. Over the course of the three health agreements, First Ministers identified priorities, including primary health care, access to care, and pharmaceuticals management.

Comparable health indicators

8.4 In the 2000 Health Communiqué, First Ministers committed to enhance accountability to Canadians through public reporting on health programs and services. The 2003 Health Accord and 2004 10-Year Plan reiterated the commitments made by First Ministers in 2000 to enhance accountability and improve performance reporting to reassure Canadians that health care reform is under way.

8.5 Specifically, the 2000 Health Communiqué directed health ministers

8.6 The 2000 Health Communiqué also directed health ministers to work with organizations across Canada with expertise in health measurement to develop common methods for measuring and reporting on the three indicator areas identified by First Ministers. According to the 2000 Health Communiqué, this collaboration would help provide for regular reporting to Canadians on the progress of health care renewal. The 2000 Health Communiqué sets out the benefits of measuring, tracking, and reporting on performance (Exhibit 8.1).

Exhibit 8.1—Benefits of measuring, tracking, and reporting on performance

Measuring, tracking and reporting on performance

Source: First Ministers' Meeting, Communiqué on Health, Ottawa, 11 September 2000

8.7 In the 2003 Health Accord, First Ministers built on their previous commitments, agreeing that each jurisdiction would continue to provide comprehensive and regular public reporting on the health care programs and services it delivers, including information on health system performance, health outcomes, and health status. They also agreed that these reports would include the health indicators set out in the 2000 Health Communiqué as well as additional comparable health indicators, to be developed by health ministers, on four themes—timely access, quality, sustainability, and health status and wellness. Jurisdictions were also to develop the necessary infrastructure to collect the data needed for quality reporting.

8.8 The 2003 Health Accord particularly noted the serious health challenges faced by Aboriginal Canadians. The federal government committed to increase its funding and cooperate with other governments and Aboriginal peoples to meet the objectives defined in the Accord. The 2003 Health Accord directed health ministers to develop, in consultation with Aboriginal peoples, a comparable Aboriginal Health Reporting Framework to inform Canadians about progress achieved on Aboriginal health outcomes.

8.9 First Ministers agreed in 2004 on a 10-year plan to strengthen health care based on a number of principles, including continued health care system accountability and the provision of information to make progress transparent to their citizens. While the 2004 10-Year Plan does not explicitly refer to comparable health indicator reporting, it committed health ministers to report to their residents on health care system performance. Health Canada sees its comparable health indicators report as one way to meet this commitment.

8.10 Comparable health indicators allow for comparisons over time or across health care systems—for example, from province to province. They provide important information to the public and the health care sector, and can play a key role in performance management, monitoring, and quality improvement. Comparable health indicators also help policy makers set priorities, plan strategically, and allocate resources.

Healthy Canadians—Health Canada's response to the First Ministers' comparable health indicator reporting commitments

8.11 The First Ministers' health agreements call for governments to demonstrate accountability through comprehensive and regular public reporting. One of the key commitments is for the federal, provincial, and territorial governments to report to the public on comparable health indicators. Health Canada has responded on behalf of the federal government by preparing Healthy Canadians: A Federal Report on Comparable Health Indicators. Healthy Canadians is published every two years, with editions in 2002, 2004, and 2006, and another scheduled for 2008. The reports present data on a number of health indicators for the general population. They also include data on certain populations for whom the federal government delivers health care services, such as First Nations and Inuit, and Canadian military personnel. The purpose of Healthy Canadians is to provide Canadians with the most current information available on comparable health indicators that measure progress on health care renewal.

Federal responsibility for health care services

8.12 While the provinces and territories are largely responsible for the delivery of health care services, the federal government supports the publicly funded system through transfer payments and the Canada Health Act. The Act is intended to ensure that all Canadians have access to medically necessary services regardless of their ability to pay. The 2006 edition of Healthy Canadians notes that the federal government directly provides some health care services and benefits to several population groups, which include

This makes the federal government one of the largest providers of health care services to Canadians.

Roles and responsibilities for health information and reporting

8.13 Health Canada is the federal department responsible for helping Canadians maintain and improve their health. To carry out this responsibility, Health Canada develops, implements, and enforces regulations, legislation, policies, programs, services, and initiatives and works with the provinces and territories, and other partners. One of the Department's objectives is to provide health information to help Canadians make informed decisions.

8.14 In addition to Health Canada, two other federal organizations are responsible for generating, managing, and reporting health information. Statistics Canada is mandated to provide accurate, timely, and relevant information about the health of Canadians and the health care system. The Public Health Agency of Canada is responsible for national health surveillance systems focusing on various communicable and chronic diseases, such as HIV and diabetes.

8.15 The Canadian Institute for Health Information (CIHI), a national, not-for-profit organization, is also responsible for managing and reporting health information. CIHI receives most of its funding from the federal government. CIHI's mandate is to coordinate the development and maintenance of a comprehensive and integrated national approach to health information.

8.16 Although Statistics Canada and CIHI prepare reports on health indicators, their reports are not a response to the First Ministers' commitments.

Recent developments related to the health agreements

8.17 In 2005, Royal Assent was granted to federal legislation authorizing a parliamentary review of the implementation of the 2004 10-Year Plan to Strengthen Health Care. The review began in March 2008. Although its focus was the progress achieved in implementing health care renewal initiatives, the review also looked at health indicator reporting. The parliamentary review, completed in June 2008, noted that the lack of standardized, uniform, and nationally comparable health data was a barrier to monitoring and reporting on the implementation of the 2004 10-Year Plan.

8.18 As part of the 2003 Health Accord, Canada's First Ministers established the Health Council of Canada, a non-profit agency funded by Health Canada ($13.8 million from 2004–05 to 2006–07). The Health Council's mandate is to report to Canadians on progress made by the federal, provincial, and territorial governments in implementing the health care agreements, particularly the accountability and transparency provisions.

8.19 In June 2008, the Health Council of Canada released its five-year progress report on what the governments have done to implement the health care renewal objectives in the 2003 First Ministers' Accord on Health Care Renewal. The report assessed the pace and direction of the renewal objectives—timely and equitable access to health care, a higher quality of care, a solid future for the public health care system, a healthier population, and more accountability to Canadians for where their money goes and what it achieves. Regarding the comparable health indicator reporting provisions of the 2003 Health Accord, the Health Council noted that only the federal government reported on comparable health indicators in 2006. The Health Council observed that a great deal of reporting on the state of health care in Canada has been produced by all governments in the past five years. Yet it noted that without more standardized and collaborative reporting by all governments—federal, provincial, and territorial—Canadians could not be confident that the new money and new practices intended to improve health care are making a difference.

Focus of the audit

8.20 The objective of our audit was to determine the extent to which Health Canada met the health indicator reporting commitments set out in the 2000, 2003, and 2004 First Ministers' health agreements, and to determine whether its reporting is improving.

8.21 Our audit examined the reporting of health indicators by Health Canada. It did not include an assessment of data quality, as this is part of our health indicator verification work (paragraph 8.23). In addition, our audit did not examine the role of provinces and territories in providing information on the health of the population and the delivery of their health care services.

8.22 More details on the audit objective, scope, approach, and criteria are in About the Audit at the end of this chapter.

8.23 Verification of the health indicators reports. In 2002, 2004, and 2006, the Office of the Auditor General was engaged by Health Canada to provide third-party verification of each edition of Healthy Canadians in accordance with a First Ministers' commitment in the 2000 Health Communiqué. We audited the health indicators to determine whether they were complete, accurate, and adequately disclosed. The definitions of these criteria were quite specific and limited. For example, accuracy was defined in terms of how well the health indicators reflected the facts and the extent to which they were reported at an appropriate level of accuracy to enable comparisons among the various editions. We concluded that the health indicators presented in each edition of Healthy Canadians met the three criteria. We were not responsible for assessing the performance achieved or the relevance or sufficiency of the health indicators selected for reporting.

Observations and Recommendations

Reporting on health indicators

Health Canada has met the specific health indicator reporting obligations

8.24 We expected Health Canada to have identified and taken action to meet its health indicator reporting obligations resulting from the First Ministers' 2000, 2003, and 2004 health agreements. We further expected that Health Canada would work with its counterparts in the provincial and territorial governments to develop a framework using jointly agreed upon comparable health indicators. Finally, we expected that Health Canada would have worked with data providers to obtain the necessary information for the health indicators reports.

8.25 We found that Health Canada identified its health indicator reporting obligations, worked with its provincial and territorial counterparts to develop a comparable health indicator framework, collaborated with data providers to obtain the necessary information for the health indicators reports, and produced health indicators reports that meet the specific health indicator reporting commitments in the health agreements.

8.26 The three health agreements were the product of federal–provincial–territorial collaboration. They included important accountability and reporting commitments, although they did not detail how the commitments were to be carried out. As part of our audit, we asked the Department for documentation outlining how it had determined its obligations in relation to the commitments made in the agreements—such documentation is important for internal management and external accountability.

8.27 Health Canada informed us that no such documentation exists. In the Department's view, the 2000 Health Communiqué and subsequent agreements clearly set out what needed to be done. The Department informed us that Healthy Canadians is its response to the First Ministers' commitments to report on comparable health indicators. However, the Department has not made clear what it is trying to achieve in publishing Healthy Canadians and how it relates to the Department's broader goals, beyond simply complying with the First Ministers' agreements.

8.28 The Department further informed us that Healthy Canadians is not intended to be the sole means to fulfill the First Ministers' commitment that health ministers provide comprehensive and regular public reporting on health programs and services they deliver, on health system performance, and on progress toward health care renewal priorities. Health Canada told us this commitment is also fulfilled through its existing publications, such as its Report on Plans and Priorities, Departmental Performance Report, and Canada Health Act Annual Report.

8.29 For the 2002 and 2004 health indicators reports, as per the health agreements, Health Canada, as part of a federal–provincial–territorial working group, worked with ministries of health and consulted with the Canadian Institute for Health Information (CIHI), Statistics Canada, and l'Institut de la statistique du Québec to develop a comprehensive framework of health indicators for comparable reporting. The health indicator framework developed for reporting in 2002 included 67 jointly agreed upon comparable health indicators, 58 of which were reported in the 2002 federal report. According to Health Canada, the remaining nine health indicators were not reported because national data were not available.

8.30 In the 2003 Health Accord, First Ministers committed to develop additional health indicators and to review those that had been developed to ensure that they were measuring progress on achieving the reforms set out in the 2003 Health Accord. This work resulted in the selection of 70 health indicators, of which 18 were chosen for detailed reporting by all jurisdictions in 2004. All 70 health indicators were not reported on because some did not meet certain criteria (for example, comparability or availability of data).

8.31 In 2006, the federal–provincial–territorial arrangement that had been in place for the two previous reports was no longer in place. In the absence of this arrangement, Health Canada reported on the same 18 health indicators chosen by the jurisdictions for reporting in 2004, as well as three additional health indicators—self-rated mental health, self-perceived stress, and self-reported fruit and vegetable consumption (Exhibit 8.2). In choosing the additional health indicators, Health Canada consulted internally and with the Public Health Agency of Canada. The three additional health indicators were selected for their relevance to current health issues and for the availability of reliable data across provinces and territories.

Exhibit 8.2—Health indicators reported in Healthy Canadians 2006

Health Indicators

Healthy Canadians 2006 was released in December 2006. It primarily included self-reported data* on health indicators for the general public from Statistics Canada's Canadian Community Health Survey. In also included data on ambulatory care sensitive conditions and diabetes indicators from CIHI and the Public Health Agency of Canada, respectively.

Theme: Timely Access

1. Self-reported difficulty obtaining routine or ongoing health care services

2. Self-reported difficulty obtaining health information or advice

3. Self-reported difficulty obtaining immediate care

4. Self-reported prescription drug spending as a percentage of income

5. Self-reported wait times for diagnostic services

Theme: Quality

6. Hospitalization rate for ambulatory care sensitive conditions

7. Self-reported patient satisfaction with overall health care services

8. Self-reported patient satisfaction with community-based care

9. Self-reported patient satisfaction with telephone health line or tele-health services

10. Self-reported patient satisfaction with physician care

11. Self-reported patient satisfaction with hospital care

Theme: Health Status and Wellness

12. Health adjusted life expectancy (HALE)

13. Prevalence of diabetes

14. Self-reported health

15. Self-reported teenage smoking rates

16. Self-reported physical activity

17. Self-reported body mass index

18. Self-reported immunization for influenza, aged 65 plus (flu shot)

19. Self-rated mental health

20. Self-perceived stress

21. Self-reported fruit and vegetable consumption

*Self-reported data are subject to some known limitations. Self-reported information requires the respondent to be honest with interviewers and to accurately recollect past events. In addition, self-reported data may be influenced by factors such as the respondent's sociodemographic characteristics, cognitive ability or memory, questionnaire design or the mode of data collection (for example, whether respondents were interviewed by phone or in person). (Return)

Source: Healthy Canadians: A Federal Report on Comparable Health Indicators 2006

8.32 Statistics Canada, CIHI and, starting in 2006, the Public Health Agency of Canada all provide Health Canada with health indicator data for Healthy Canadians.

8.33 According to the Health Council of Canada, Health Canada, and the provinces and territories published comparable health indicators reports in 2002 and 2004. In 2006, however, only Health Canada published a comparable health indicators report.

8.34 The First Ministers' health agreements called for enhanced accountability to the public on health care renewal through improved performance reporting. This was to help reassure Canadians that health care reforms are occurring. While Health Canada has met the specific comparable health indicator reporting commitments, as we note in the sections that follow, it has not fulfilled the broader intent of the agreements—to inform Canadians on the progress of health care renewal.

Improvement over time

Healthy Canadians lacks interpretation of progress on health care renewal

8.35 While Health Canada met the federal government's specific health indicator reporting obligations, a key goal of the three health agreements was to report to Canadians on the progress of health care renewal, including accessibility, quality, and sustainability of the publicly funded health care system. One way that reporting on health care renewal is done is through health indicators. Good public reporting would state why the information reported is significant and relevant, and explain why trends may be occurring. Interpretation is necessary to link individual health indicators to overall progress. Interpretation of the information in Healthy Canadians could help readers see what progress the government is making in health care renewal in relation to investments pledged for health care reform.

8.36 In this audit, we found that the health indicators reports presented data on health indicators without providing sufficient interpretation (See Exhibit 8.3 for an example). The reports present health indicators in isolation, without sufficient explanation of their individual significance or how, taken together, they reflect progress on health care renewal.

Exhibit 8.3—Healthy Canadians 2006 presentation of self-reported patient satisfaction with physician care indicator

Sample page from the 2006 report Healthy Canadians: A Federal Report on Comparable Health Indicators

[text version]

Source: Healthy Canadians: A Federal Report on Comparable Health Indicators 2006

8.37 The lack of explanation and discussion in all three editions of Healthy Canadians can also be seen in how they present self-reported wait times for diagnostic services. The 2002, 2004, and 2006 editions provide a definition of each indicator in terms of what it measures, and they present indicator data both textually and graphically. However, apart from one sentence in the overview section of the 2004 and 2006 editions—about the advantages of early access to diagnostic equipment for disease detection, diagnosis, and treatment—no interpretive information is given on wait times. Interpretive information could help readers to have a better sense of what progress has been made in reducing wait times.

8.38 In contrast is Saskatchewan's Comparable Health Indicators Report 2004—the province's equivalent of the federal Healthy Canadians. Unlike the federal Healthy Canadians, this provincial report provides interpretive information about barriers to accessing care that might influence wait times—for example, the availability of equipment and facilities or the number of radiologists responsible for operating the equipment. The report also cites data about the most common consequences experienced by patients of waiting for diagnostic tests. As well, it provides a description of wait time projects with the associated financial details and web links for further information. This interpretive information helps the reader understand the significance of the health indicator.

8.39 Interpretation is also provided in the 2007 Canadian Institute for Health Information's report, Health Care in Canada. This report presents selected health indicators in context to tell a compelling performance story. For example, the report situates access to care in light of what is known and unknown in terms of how long people wait and how waits affect their health and well-being. The report also tracks the number of MRI and CT scanners available and provides information on the number of scans performed in Canada compared to other countries. Much like Saskatchewan's comparable health indicators report, the 2007 CIHI report provides information on the consequences of waiting for diagnostic tests, as reported by patients. The report also provides a web link to the CIHI site for further information on access and wait times.

8.40 The Public Health Agency of Canada's Report on the State of Public Health in Canada 2008, issued by the Chief Public Health Officer, is also an example of a federal report that includes interpretation of health indicators—such as using education and income to help explain differences in health status.

8.41 Our findings that the Healthy Canadians reports lack interpretation are consistent with feedback that Health Canada received in 2003, when focus group participants indicated that Healthy Canadians would be more meaningful if it included interpretive information.

8.42 Recommendation. For future editions of Healthy Canadians, Health Canada should include interpretive information to help Canadians understand what the health indicator data mean as well as how health care renewal is progressing.

The Department's response. Agreed. In earlier versions of Healthy Canadians, Health Canada focused on presenting data on comparable indicators clearly and unambiguously, avoiding interpretations that were not supported by the data alone and which might have been challenged by alternative perspectives. For the 2008 Healthy Canadians report, Health Canada will continue to emphasize data validity and reliability but is also exploring ways of interpreting the data that would make it more meaningful to Canadians. A Committee has been established with a mandate to provide advice and guidance on the development of the 2008 Healthy Canadians report and, in particular, on ways of presenting the data with more interpretive analysis, while at the same time retaining any necessary cautionary and explanatory notes about data validity and reliability. The Committee, which met for the first time on January 31, 2008, includes members from Health Canada and the Public Health Agency of Canada.

Health Canada does not know if recent editions of Healthy Canadians meet the needs of Canadians

8.43 The First Ministers' health agreements called for comparable health indicators reports that would be informative and relevant to the public. In meeting this commitment, we expected Health Canada to consult with Canadians about their information needs.

8.44 In 2003, Health Canada commissioned focus group testing in five major centres across Canada to determine how well the 2002 edition of Healthy Canadians responded to the needs of the public. Focus group participants indicated that the report was easy to read and understand, and written in clear, straightforward language. They found the graphs clear and useful, and the layout of the report easy to scan.

8.45 What the focus group participants liked least was the lack of explanations or interpretation of the information in the report. The focus group study notes, "As a result, the statistics tended to blend together without meaning or definition for readers." Focus group participants called for future editions of Healthy Canadians to add interpretive language that would provide greater meaning and context for the information presented. Participants were most interested in the interpretation of trends, explanations of causes and effects, and information that they could personally act on.

8.46 In both 2003 and 2004, Health Canada (prior to issuing the 2004 edition of Healthy Canadians), as part of a federal–provincial–territorial working group, conducted a number of consultations to guide their selection of health indicators for comparable reporting. In 2003, the working group consulted with experts and stakeholders through an invitational workshop as well as through written submissions and via the Internet. In 2004, the working group consulted with the public through focus group testing to determine the health indicators that were important and meaningful to Canadians, as well as to provide information on the public's preferences for the format and presentation of the health indicators.

8.47 To date, Health Canada has not consulted with the public on its 2004 and 2006 editions of Healthy Canadians. Canadians' information needs may have changed since the 2003 and 2004 consultations. The Department attempted to obtain feedback on the 2006 edition by including in it a reader feedback questionnaire. However, very few people have responded. During our audit, Health Canada announced plans for a 2008 evaluation of Healthy Canadians to determine whether it has met its original objectives. The Department planned to consult with previous recipients of federal reports, members of federal populations, individuals who took part in earlier consultations to select comparable health indicators, and key Departmental representatives. The final evaluation report was originally to be completed in September 2008, but was rescheduled for reporting in August 2009.

8.48 Recommendation. In planning for future editions of Healthy Canadians, Health Canada should take into account the results of its consultations and its 2009 evaluation to determine the needs of Canadians and how Healthy Canadians can best meet them.

The Department's response. Agreed. Based on previous consultations, Health Canada is looking at ways to improve interpretation of the data in the 2008 Healthy Canadians report, and is considering the relevance of the indicators reported. Health Canada is also planning an evaluation of the 2008 Healthy Canadians, including input from external stakeholders such as health interest groups, which will be completed by August 2009, in time to incorporate findings into future federal reports.

Healthy Canadians has not improved over time

8.49 We expected to find improvements in successive health indicators reports—that is, provision of additional or updated data and health indicators, and better presentation of health indicators. In addition, we expected Health Canada to use the feedback from the consultations they carried out to improve subsequent reports.

8.50 We compared the 2004 and 2006 editions of Healthy Canadians with the 2002 edition to determine whether the Department had made improvements in providing additional or updated data and health indicators and in presenting the information.

8.51 In 2004, Health Canada reported on 18 featured health indicators. In its 2006 edition of Healthy Canadians, Health Canada reported on 21 health indicators—the same ones it reported on in 2004 plus three new health indicators—self-rated mental health, self-perceived stress, and self-reported fruit and vegetable consumption. In choosing the additional health indicators, Health Canada consulted internally and with the Public Health Agency of Canada. The three health indicators were selected for their relevance to current health issues and for the availability of reliable data across provinces and territories. These three health indicators were not part of the framework of 70 health indicators approved for comparable reporting by the Conference of Deputy Ministers. Health Canada has indicated that in recognition of provincial and territorial responsibilities for health care and the commitments made in the health agreements, it is important for the Department to consult with provincial and territorial governments on future changes to health indicators.

8.52 Apart from the addition of these three health indicators, we found that the presentation of the information in 2002 was largely unchanged in 2004 and 2006. The reports for those later years continued to provide data on health indicators without adequate explanation and discussion of their implications (paragraphs 8.36 and 8.37).

8.53 As noted earlier, in an attempt to improve Healthy Canadians, Health Canada consulted with Canadians in 2003. The public provided feedback on the extent to which the 2002 report met their information needs (paragraphs 8.44 and 8.45). However, we found that Health Canada did not prepare an action plan responding to the concerns expressed in the consultations and that the 2004 and 2006 editions of Healthy Canadians did not address key deficiencies highlighted through the public consultation.

8.54 Recommendation. In planning for future editions of Healthy Canadians, Health Canada should review the health indicators reported on and take concrete steps to improve their presentation.

The Department's response. Agreed. Health Canada has reviewed the current list of comparable health indicators, which was developed via a federal–provincial–territorial consultation that ended in June 2005. This review, completed in September 2008, assessed the extent to which these indicators reflect current concerns of Canadians and ongoing commitments of the Health Accords and has been used in the development of Healthy Canadians 2008.

Reporting on federal populations is inconsistent across editions of Healthy Canadians

8.55 As noted earlier (paragraph 8.12), the federal government is directly responsible for providing some health care services to several population groups. These include First Nations and Inuit; military personnel and veterans; inmates of federal correctional facilities; members of the RCMP; and asylum seekers, refugees, and persons detained for immigration purposes. For populations other than First Nations and Inuit, other federal departments apart from Health Canada are responsible for delivering some health care services.

8.56 Healthy Canadians reports include health indicators on only some of these populations. Health Canada notes in Healthy Canadians that this is because data are sometimes lacking and because federal departments and agencies do not always have the capacity to collect the data. Moreover, available data on these federal populations often do not correspond to the requirements of comparable indicator reporting (Exhibit 8.4).

Exhibit 8.4—Healthy Canadians cover federal populations inconsistently

Reports Populations included in the report Populations not included in the report

2002

  • First Nations and Inuit
  • military personnel
  • federal inmates
  • veterans
  • asylum seekers, refugees, and persons detained for immigration purposes
  • RCMP

2004

  • First Nations and Inuit
  • military personnel
  • veterans
  • federal inmates
  • asylum seekers, refugees, and persons detained for immigration purposes
  • RCMP

2006

  • First Nations and Inuit
  • military personnel
  • veterans
  • federal inmates
  • asylum seekers, refugees, and persons detained for immigration purposes
  • RCMP

8.57 We examined whether Health Canada had assessed the merit, applicability, and feasibility of reporting on the federal populations in Healthy Canadians. We found that Health Canada had not done so. As Health Canada considers how to address the results of its 2009 evaluation of Healthy Canadians, it would be an ideal time to consult with the federal departments responsible for these populations regarding their inclusion in future editions of Healthy Canadians. Through these consultations with the other departments, Health Canada can demonstrate its leadership and commitment to reporting on health indicators.

8.58 Recommendation. Health Canada should consult with the departments responsible for federal populations regarding the merit, applicability, and feasibility of including information on these populations in future editions of Healthy Canadians.

The Department's response. Agreed. In 2008, Health Canada plans to report any available new data in relation to health services delivered by other federal departments to specific population groups (National Defence, Veterans Affairs Canada, Citizenship and Immigration Canada, Correctional Service Canada, Royal Canadian Mounted Police). For Healthy Canadians reports in 2010 and beyond, Health Canada plans to consult with the other departments on the merit, applicability, and feasibility of reporting on these populations.

Limited First Nations and Inuit health indicator information is reported in Healthy Canadians

8.59 Health Canada provides a range of health care services to First Nations and Inuit—the largest federal population. We reviewed the First Nations and Inuit health indicators reported in the 2002, 2004, and 2006 editions of Healthy Canadians to determine what health indicators were reported and how they were presented.

8.60 The three editions of Healthy Canadians included limited First Nations and Inuit health indicators. It should be noted that Healthy Canadians is not intended to be the sole vehicle for reporting on First Nations and Inuit by Health Canada. For the 2002 edition of Healthy Canadians, Health Canada reported on a subset of the health indicators from the indicator framework and much of the data came from Health Canada itself. For the 2004 and 2006 editions of Healthy Canadians, much of the data Health Canada reported was supplied by the Aboriginal Peoples Survey and the First Nations Regional Longitudinal Health Survey, respectively.

8.61 Not all of the health indicators chosen for reporting on the general population have corresponding health indicators for the First Nations and Inuit population. Reporting on First Nations and Inuit health is challenging because it is difficult not only to develop culturally sensitive reporting frameworks, but also to carry out efficient and effective ways of data collection. This population is dispersed across many regions of remote Canada as well as urban centres.

8.62 Aboriginal Health Reporting Framework. In the 2003 Health Accord, First Ministers recognized the serious challenges that face the health of Aboriginal Canadians, including First Nations and Inuit. Disparity in the health status of Aboriginal peoples compared to the overall Canadian population is significant (Exhibit 8.5). First Ministers agreed to work together to address the gap in health status between Aboriginal and non-Aboriginal Canadians.

Exhibit 8.5—Disparity in health status between First Nations and Inuit and the Canadian population

8.63 First Ministers directed health ministers to consult with Aboriginal peoples on the development of a comparable Aboriginal Health Reporting Framework (the Framework) and to develop the necessary data infrastructure to report to Canadians and Aboriginal peoples on progress achieved and key outcomes of health care renewal. In the 2003 Health Accord, the federal government committed to working collaboratively with other governments and the Aboriginal peoples in this regard.

8.64 To respond to the First Ministers' commitment to develop the Framework, a task group was established in 2004. The task group consisted of representatives from the federal government (including Health Canada), the provinces and territories, and five national Aboriginal organizations. However, in 2005, the work of the task group was suspended and only in 2008 did work resume on the Framework. It will be important for the remaining work to be completed to help address the disparity in health status between Aboriginal peoples and the general population to the extent that work on this Framework can contribute to the development of indicators that can be reported in Healthy Canadians and other reports.

8.65 Recommendation. Health Canada should continue to develop First Nations and Inuit health indicators in collaboration with other governments and Aboriginal peoples, and report these indicators in future editions of Healthy Canadians, to the extent that it is practicable.

The Department's response. Agreed. Health Canada continues to collaborate with First Nations and Inuit stakeholders and provincial and territorial partners on a number of initiatives aimed at developing and collecting data on relevant indicators. Health Canada plans to report on new data and indicators in the 2010 Healthy Canadians report, based on an update from the Federal/Provincial/Territorial Task Group on Aboriginal Health Data and Indicators (December 2008); the initial release of the 2006 Off-Reserve Aboriginal Peoples Survey on October 29, 2008; and, anticipated results in fall 2009 from the second cycle of the First Nations Regional Longitudinal Health Survey. Given the current timelines for the 2008 Healthy Canadians report, Health Canada will report on a limited number of indicators in this report.

Healthy Canadians receives little media attention

8.66 A communications strategy is an important tool for increasing the exposure of a publication. It can include analysis of how information will be disseminated to stakeholders, such as the media and other target audiences, and can involve working with stakeholders before the publication's release date to promote maximum exposure. If Healthy Canadians is to inform Canadians, Canadians need to be aware it exists. We expected Health Canada to have developed a communications strategy for promoting Healthy Canadians.

8.67 Although Health Canada developed communications strategies for the 2004 and 2006 editions of Healthy Canadians, we found very few references to either report in the media. In contrast, other reports, including other health indicators reports, that are frequently cited by the national media are produced by organizations that work extensively with the media in advance of report release dates to help ensure that their reports receive coverage.

8.68 Recommendation. In planning for future editions of Healthy Canadians, Health Canada should prepare a communications strategy designed to ensure that Canadians are aware of the reports.

The Department's response. Agreed. Health Canada developed communications strategies for previous Healthy Canadians reports, and will do so for the 2008 Healthy Canadians report. As part of its overall communications strategy, it will examine various ways to make Canadians aware of the report and its contents, including posting it on the Health Canada website and sending announcements of its release to health organizations and health professionals.

Health Canada's position on the Health Council of Canada's recommendations is unclear

8.69 Since 2005, the Health Council of Canada has released annual reports that provide a national system-wide perspective on progress made by the federal, provincial, and territorial governments in implementing the health care agreements, including the accountability and transparency provisions. The Health Council of Canada has made a number of recommendations with respect to comparable health indicator reporting, including that health ministers revisit their approach to producing health indicators reports, noting that, to date, comparable health indicators reports have not been a useful reporting mechanism for the general public. The Health Council has stated that, if governments are clear about their purpose, the comparable health indicators reports can become a useful public tool for assessing health care system improvement. It recommended linking reporting strategies to health care goals and including socio-economic factors in the health indicator framework.

8.70 While the Health Council's recommendations are directed to all jurisdictions and are not binding, in that a formal response to the recommendations is not required, we expected Health Canada to have analyzed the Health Council's recommendations to determine what action, if any, it should take. We found that Health Canada did analyze successive Health Council reports. However, we did not find evidence of what actions the Department decided to take in response to the recommendations.

Health Canada needs to devise a strategy to improve Healthy Canadians

8.71 We have made a number of observations that question the value of Healthy Canadians. As noted earlier, successive editions of Healthy Canadians are not receiving attention from the media and public. They remain largely unchanged since the first edition in 2002. Without interpretation, their ability to inform Canadians is limited. Health Canada needs to clarify the role of Healthy Canadians relative to other health indicators reports produced by CIHI, Statistics Canada, and the Chief Public Health Officer. It also needs to determine what it is trying to achieve above and beyond simply complying with the First Ministers' agreements, and how Healthy Canadians links to Health Canada's broader goals.

8.72 Given that three editions of Healthy Canadians have been released and we have found that the reports have not improved over time, it is an opportune time for Health Canada to conduct a thorough review of its role and its approach to health indicator reporting. Options could include working with leading organizations with expertise in health reporting to better identify Canadians' needs and to improve analysis and reporting strategies.

8.73 Recommendation. Based on a thorough review of its role and approach to health indicator reporting, Health Canada should set out an approach to improve the Healthy Canadians reports, meet the health indicator reporting requirements in the health agreements, and ensure their relevance and usefulness to Canadians. The Department may also want to consider how best to capitalize on the expertise of other national organizations to improve future editions of Healthy Canadians.

The Department's response. Agreed. Health Canada will evaluate its approach to health indicator reporting under the First Ministers' Accords following the release of Healthy Canadians 2008. In its evaluation, Health Canada will consult with national organizations on how their expertise can contribute to improving the Healthy Canadians 2010 report and beyond.

Conclusion

8.74 Healthy Canadians is Health Canada's response to First Ministers' commitments in the 2000, 2003, and 2004 health agreements to report on comparable health indicators. We reviewed the 2002, 2004, and 2006 editions of Healthy Canadians to determine the extent to which they met the commitments. We found that although Health Canada met the specific commitments to report on health indicators, the Department did not do so in a way that fulfilled the broader intent of the agreements—to provide information to Canadians on the progress of health care renewal.

8.75 The reports present data on a number of health indicators without providing sufficient interpretation. Health indicators are presented in isolation, without clear explanation of their individual significance or how, together, they reflect progress on health care renewal. They do not tell a performance story.

8.76 We also reviewed each edition of Healthy Canadians to see if it had improved over time. We found that the presentation of the information in all three editions was essentially the same, with some modest improvements. Each edition continued to provide data on health indicators without adequate explanation and discussion of their implications—despite the fact that Health Canada had received feedback through consultations with Canadians indicating that their information needs were not being met through the reports.

8.77 Healthy Canadians has the potential to be a useful tool in informing Canadians about their health and about health care system renewal. However, Health Canada must provide readers with interpretation of the comparable health indicators and link them to progress in health care renewal. Given that Health Canada has produced three editions of Healthy Canadians and will be reporting its evaluation of the 2008 Healthy Canadians in August 2009, it is an opportune time for Health Canada to conduct a thorough review of its role and its approach to health indicator reporting.

About the Audit

Objective

The objective of our audit was to determine the extent to which Health Canada met health indicator reporting commitments, including those in the First Ministers' health agreements, and to determine whether its reporting is improving.

Scope and approach

In our audit, we examined how Health Canada reported on health indicator information. We did not examine issues of data quality, or the role of provinces and territories in providing information on the health of their populations and the delivery of health care services. Also excluded from our audit were international reporting requirements and health indicators reported on the Internet.

Criteria

Listed below are the criteria that were used to conduct this audit and their sources.

Criteria Sources

We expected Health Canada's efforts to produce health indicators reports to have involved identifying its health indicator reporting commitments under the First Ministers' health agreements, as well as organizing its activities to meet these commitments. This includes establishing the relationships required to enable Health Canada to meet its commitments.

  • September 2000 First Ministers' Meeting Communiqué on Health, Clear Accountability—Reporting to Canadians
  • 2003 First Ministers' Accord on Health Care Renewal, Reporting to Canadians on Change, and Annex A to Accord
  • First Ministers' Meeting on the Future of Health Care 2004, A 10-year plan to strengthen health care, Accountability and Reporting to Citizens
  • Treasury Board of Canada Secretariat, Management Accountability Framework, Areas of Management (2007), Governance and Strategic Directions, Section 4.3

To meet its health indicator reporting commitments, we expected Health Canada to have identified the target audience for the Healthy Canadians reports, determined the needs of this audience, and monitored the extent to which these needs were being met.

  • September 2000 First Ministers' Meeting Communiqué on Health, Clear Accountability—Reporting to Canadians
  • 2003 First Ministers' Accord on Health Care Renewal, Reporting to Canadians on Change, and Annex A to Accord
  • First Ministers' Meeting on the Future of Health Care 2004, A 10-year plan to strengthen health care, Accountability and Reporting to Citizens
  • CCAF-FCVI, What can we learn from effective public performance reporting, Good Practices for Central Agencies, Legislators, Auditors, and Report Producers (2007), Good Practice: Consult Users

To meet its health indicator reporting commitments, we expected Health Canada to have improved its reporting of health indicators over time by identifying gaps between the needs of the identified target audience and its current reporting, and to have worked to fill such gaps.

  • September 2000 First Ministers' Meeting Communiqué on Health, Clear Accountability—Reporting to Canadians
  • 2003 First Ministers' Accord on Health Care Renewal, Reporting to Canadians on Change, and Annex A to Accord
  • First Ministers' Meeting on the Future of Health Care 2004, A 10-year plan to strengthen health care, Accountability and Reporting to Citizens
  • Treasury Board of Canada Secretariat, Management in the Government of Canada: A Commitment to Continuous Improvement (2005), Section 2

Audit work completed

Audit work for this chapter was substantially completed on 15 June 2008.

Audit team

Assistant Auditor General: Neil Maxwell
Principal: Glenn Wheeler
Lead auditor: Doreen Deveen

Irene Andayo
Ruth Sullivan

For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free).

Appendix—List of recommendations

The following is a list of recommendations found in Chapter 8. The number in front of the recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses indicate the paragraphs where the topic is discussed.

Recommendation Response
Improvement over time

8.42 For future editions of Healthy Canadians, Health Canada should include interpretive information to help Canadians understand what the health indicator data mean as well as how health care renewal is progressing. (8.35–8.41)

Agreed. In earlier versions of Healthy Canadians, Health Canada focused on presenting data on comparable indicators clearly and unambiguously, avoiding interpretations that were not supported by the data alone and which might have been challenged by alternative perspectives. For the 2008 Healthy Canadians report, Health Canada will continue to emphasize data validity and reliability but is also exploring ways of interpreting the data that would make it more meaningful to Canadians. A Committee has been established with a mandate to provide advice and guidance on the development of the 2008 Healthy Canadians report and, in particular, on ways of presenting the data with more interpretive analysis, while at the same time retaining any necessary cautionary and explanatory notes about data validity and reliability. The Committee, which met for the first time on January 31, 2008, includes members from Health Canada and the Public Health Agency of Canada.

8.48 In planning for future editions of Healthy Canadians, Health Canada should take into account the results of its consultations and its 2009 evaluation to determine the needs of Canadians and how Healthy Canadians can best meet them. (8.43–8.47)

Agreed. Based on previous consultations, Health Canada is looking at ways to improve interpretation of the data in the 2008 Healthy Canadians report, and is considering the relevance of the indicators reported. Health Canada is also planning an evaluation of the 2008 Healthy Canadians, including input from external stakeholders such as health interest groups, which will be completed by August 2009, in time to incorporate findings into future federal reports.

8.54 In planning for future editions of Healthy Canadians, Health Canada should review the health indicators reported on and take concrete steps to improve their presentation. (8.49–8.53)

Agreed. Health Canada has reviewed the current list of comparable health indicators, which was developed via a federal–provincial–territorial consultation that ended in June 2005. This review, completed in September 2008, assessed the extent to which these indicators reflect current concerns of Canadians and ongoing commitments of the Health Accords and has been used in the development of Healthy Canadians 2008.

8.58 Health Canada should consult with the departments responsible for federal populations regarding the merit, applicability, and feasibility of including information on these populations in future editions of Healthy Canadians. (8.55–8.57)

Agreed. In 2008, Health Canada plans to report any available new data in relation to health services delivered by other federal departments to specific population groups (National Defence, Veterans Affairs Canada, Citizenship and Immigration Canada, Correctional Service Canada, Royal Canadian Mounted Police). For Healthy Canadians reports in 2010 and beyond, Health Canada plans to consult with the other departments on the merit, applicability, and feasibility of reporting on these populations.

8.65 Health Canada should continue to develop First Nations and Inuit health indicators in collaboration with other governments and Aboriginal peoples, and report these indicators in future editions of Healthy Canadians, to the extent that it is practicable. (8.59–8.64)

Agreed. Health Canada continues to collaborate with First Nations and Inuit stakeholders and provincial and territorial partners on a number of initiatives aimed at developing and collecting data on relevant indicators. Health Canada plans to report on new data and indicators in the 2010 Healthy Canadians report, based on an update from the Federal/Provincial/Territorial Task Group on Aboriginal Health Data and Indicators (December 2008); the initial release of the 2006 Off-Reserve Aboriginal Peoples Survey on October 29, 2008; and, anticipated results in fall 2009 from the second cycle of the First Nations Regional Longitudinal Health Survey. Given the current timelines for the 2008 Healthy Canadians report, Health Canada will report on a limited number of indicators in this report.

8.68 In planning for future editions of Healthy Canadians, Health Canada should prepare a communications strategy designed to ensure that Canadians are aware of the reports. (8.66–8.67)

Agreed. Health Canada developed communications strategies for previous Healthy Canadians reports, and will do so for the 2008 Healthy Canadians report. As part of its overall communications strategy, it will examine various ways to make Canadians aware of the report and its contents, including posting it on the Health Canada website and sending announcements of its release to health organizations and health professionals.

8.73 Based on a thorough review of its role and approach to health indicator reporting, Health Canada should set out an approach to improve the Healthy Canadians reports, meet the health indicator reporting requirements in the health agreements, and ensure their relevance and usefulness to Canadians. The Department may also want to consider how best to capitalize on the expertise of other national organizations to improve future editions of Healthy Canadians. (8.69–8.72)

Agreed. Health Canada will evaluate its approach to health indicator reporting under the First Ministers' Accords following the release of Healthy Canadians 2008. In its evaluation, Health Canada will consult with national organizations on how their expertise can contribute to improving the Healthy Canadians 2010 report and beyond.

 


Definition:

Health indicator—A single summary measure, most often expressed in quantitative terms, that represents a key aspect of health status, the health care system, or related factors. Indicators rely on consistency in data collection and are used to measure progress against benchmarks.

Source: Canadian Institute for Health Information (Return)

 

PDF Versions

To access the Portable Document Format (PDF) version you must have a PDF reader installed. If you do not already have such a reader, there are numerous PDF readers available for free download or for purchase on the Internet: