B.C., Alberta share their experiences

Observing the evolution of public performance reporting in Canada is like watching a cake baking in an oven. All the ingredients are there for an excellent cake. The batter is obviously changing from its raw state to baked perfection. But you certainly wouldn't want to pull the pan out of the oven now and serve the cake to anyone as a finished product.

Over the last decade or so, Canadian jurisdictions have been moving steadily forward to strengthen public performance reporting. As CCAF's recent research report Toward Producing and Using Better Public Performance Reporting - Perspectives and Solutions demonstrated, much progress has been made, and much more remains to be accomplished.

One important element in the evolution of public performance reporting is the role of the audit function vis-à-vis performance reports. The Auditors General of Alberta and British Columbia have been leaders in exploring how legislative audit offices can encourage governments to produce useful performance reports.

CCAF discussed this subject with Susan Jennings, Assistant Auditor General in the Office of the Auditor General of British Columbia, and Ronda White, Assistant Auditor General in the Office of the Auditor General of Alberta.

CCAF: First, what role does each of your offices play in the performance reporting process?

Ronda White:
Among other roles, the Alberta Auditor General produces audit reports on the performance measures of the government and on those of all 24 ministries as included in their annual reports. In previous years we have also examined the quality of the ministry annual reports and whether they comply with Ministry Annual Report Standards established by the Department of Finance. Ministry annual reports are reviewed by the public accounts committee and we have noted a trend over the last few years of increased discussion relating to the non-financial performance measures.

Since the quality of a performance report is largely dependant on the quality of the plan, we also examine ministry business plans. We have reviewed ministry business plans to determine whether they comply with Ministry Business Plan Standards issued by the Department of Finance, and whether plans incorporate good practices.

Susan Jennings:
In British Columbia, we produce an annual assessment of the quality of the government's performance reporting. We do not, however, provide users with independent assurance that the information in these reports is relevant and reliable.

CCAF: What is the rationale for legislative audit offices to audit performance information?

Susan Jennings:
Users of a performance report will find one of four levels of assurance that the information in the report is reliable.

• At the lowest level, the report simply assumes that the information is valid.
• At the second level, an individual such as the Minister affirms that the information is reliable.
• At level three, the report describes how the information is reliable.
• And at the highest level, a third party provides independent assurance.

In BC, most annual reports are at level one or two. We think the third-party opinion we can offer assures users of performance reports that they can take the reported performance at face value and move straight to considering whether performance is acceptable or not. Without such assurance, discussions around performance will always be limited by doubts about the information used and the fairness of its presentation.

Ronda White:
Auditors can add credibility to performance information. In most cases readers of annual reports will not have access to the data used to prepare performance information, or the time, resources or expertise to verify that it is correct. There are four different ways that auditors can add credibility, or third-party assurance, as Susan describes it. We can:

• Perform specified auditing procedures on measures.
• Provide moderate assurance: is the performance information plausible?
• Provide a high level of assurance that measures are reliable, complete, comparable and understandable.
• Provide the highest level of assurance on measures, including their relevance and sufficiency.

I believe that providing assurance on performance information adds credibility to that information. And the more credible performance reports are, the more likely they are to be used.

Ronda White joined the Auditor General of Alberta in 1997. Previously, she worked with two private sector accounting firms. She was promoted to Assistant Auditor General in 2003 and is currently responsible for the provision of audit and advisory services to the ministries of Executive Council, Human Resources and Employment, Community Development, Justice and Attorney General, Seniors and Community Supports, and Solicitor General and Public Security. She is also responsible for the Office's cross-government human resource management, business planning and performance reporting audits. She is a Chartered Accountant and has a Bachelor of Commerce from the University of Alberta.

Susan Jennings joined the Office of the Auditor General of BC in 1991 after several years as a management consultant with the Department of National Defence. She increasingly focuses her attention on issues of accountability, performance management and public performance reporting in the public sector. As lead author of the Office's reports on enhancing accountability for performance, she helped develop a framework for government reporting to the Legislative Assembly.

Susan is one of a national group of legislative auditors that is developing principles and practice for reporting and auditing of non-financial performance information. She holds a Bachelor of Arts degree from Carleton University and is a Certified Management Consultant (CMC) with the Canadian Association of Management Consultants.

CCAF: What is the current status of your efforts to provide assurance on performance information?

Ronda White:
The Alberta Auditor General began a phased approach to providing assurance on performance measures in 1995. In 1995 and 1996, we applied three basic procedures to report on the reliability of the performance measures included in government annual reports. In 1997, we did a preliminary assessment of control systems, which we did not include in our report. In 2000, we expanded our procedures to include an assessment of comparability, reliability of control systems, and understandability of the performance measures. Our current report verifies that performance measures are reliable, comparable, understandable and complete in relation to the business plan. We do not provide an opinion on whether the measures are relevant and sufficient to assess the performance of the Ministry in achieving its goals.

We have also carried out audits of “managing for results systems” including business planning, performance measurement and human resources of several ministries. As part of this work we also comment on the relevance and sufficiency of measures in relation to goals in the business plan which is challenging work to carry out.

Susan Jennings:
Several years ago, British Columbia adopted a set of reporting principles that reflect broad agreement among government, legislators and the Auditor General about the basis on which government should report its performance. As we said in our most recent report, for us to provide audit assurance on all government reports would first require annual reports to fully meet the requirements of the BC Reporting Principles, and organizations to provide evidence of robust systems for the collection of performance information. Unfortunately, we are a long way from that stage of development.

This is true not just at the level of individual government organizations, but notably at the government-wide level. We used our assessment of the 2004-2005 annual reports to start focusing on the sort of information that would be required to allow readers to determine whether the government is delivering on its high-level strategic priorities.

There are some initial success stories at the level of individual organizations. The Workers' Compensation Board of BC, for example, has produced a report that we believe does present performance fairly and in accordance with the BC Reporting Principles, and we have said so. Several other Crown agencies are approaching this standard, and again we have commended them for their efforts.

CCAF: What constraints are your offices facing in providing assurance about performance information?

Ronda White:
We have found that auditing performance measures is more difficult and time consuming than we had expected. Measures and goals are subject to change. Measures relate to different time periods. Measures often come from unique and complex systems that have not historically been audited. Assessing relevance and sufficiency is also challenging.

Susan Jennings:
This is still a new field, and we have a lot to learn. We are trying to decide how much assurance we should provide; whether we should assess all annual reports; whether we should look at the whole report or part of it; and whether we should report on relevance and/or accuracy.

CCAF: Where do you go from here?

Ronda White:
The Alberta AG's Office is evaluating the state of performance measurement and reporting in government and considering the level of assurance required. We are asking the same kinds of questions BC is asking. Is performance information used, and does the user require assurance? If so, what level of assurance is required? Should we assess relevance and sufficiency? If so, how and when? Is assurance required on all performance information and reports? What is the cost-benefit, for both preparer and auditor, of providing assurance on performance information? We are also reviewing out audit approach to performance measures to determine more efficient ways of doing this work.

A critical issue is who are the users of the information and what level of assurance do they require?

Susan Jennings:
We have recommended that meeting the reporting principles be made a legislated requirement for all government organizations, as an important step in strengthening performance reporting.

But we too are conscious of the fact that these reports are produced for users outside of the government management community, particularly legislators and the public. We intend now to assess the use made of government performance reports by legislators, to determine whether the current reporting meets their needs and how it could be improved.

We're also looking forward to working through CCAF's Sloan Program to see how we can encourage improved public performance reporting and increased use of performance reports in British Columbia.