E-guide On Participatory Audit

What Success Looks Like and How to Measure Impact

  1. Module 04
  2. Outcomes: Impact on overall governance and accountability system

Outcomes: Impact on overall governance and accountability system

Tracing outcomes deserves a closer look to understand the appropriateness of the mechanisms selected and the conditions in which SAI–citizen engagement works, as well as its overall impact on accountability and improved governance. Measuring outcomes is a particularly challenging task, especially in initiatives such as SAI–citizen engagement in which the main outcomes deriving from coalition building and improved capacity are, more often than not, intangible. For that reason, an important activity is to carefully document processes over time and see how they contribute to achieving the goals set at the beginning of the engagement. To that end, goals must be measurable and articulated in a clear, concise manner.

Citizen Engagement to Improve Audit Outcomes

Inputs Outputs Intermediate
Outcomes
Citizen Engagement in the external Audit Process
  • Capacity-building support to initiate the dialogue between SAIs and CSOs and raise awareness of possible entry points for collaboration (planning, execution, dissemination, and follow-up)
  • Technical support for SAI to develop and implement a citizen-engagement strategy (such as a citizen desk, a dedicate website, and hotlines)
  • Capacity-building support to mainstream citizen engagement in the audit planning, fieldwork during performance audits, simplification (citizen audit reports) and dissemination of audit reports and recommendations, and monitoring the status of audit
    • recommendations.
  • Capacity building and technical support for a citizen-friendly communication strategy.
  • Engagement strategy finalized, cleared, and under implementation by SAIs.
  • Citizens providing input into the audit-planning process
  • SAIs conducting joint audits with CSOs ( use of social accountability tools for data collection in performance audits)
  • SAIs conducting public consultation meetings at the national or local level for dissemination of the past year’s reports and seeking feedback for current cycle
  • SAIs producing citizen audit reports
  • Increased number of CSOs and media monitoring the status of implementation of audit recommendations by the government
  • Increased quality and timeliness of audit reports
  • Improved follow-up action to audit findings because of CSOs’ interventions
  • Increased public trust in SAIs, which became more inclusive

Because outcomes in this area follow a nonlinear trajectory, a robust methodology is required to collect information to help understand the different change paths affecting both direct and indirect outcomes. Such an approach is necessary to not only understand how activities create outputs and then outcomes but also to grasp how the process of SAI–citizen engagement is influenced by different actors pursuing different agendas and goals.

Direct Outcomes

Measuring and mapping the direct outcomes resulting from activities implemented in the short and medium term is critical to evaluating the appropriateness and effectiveness of the SAI–citizen engagement mechanisms implemented in meeting the expected goals. Usually, the excepted outcomes resulting from SAI–citizen engagement would include more responsive and accountable states, new spaces for citizen engagement, improved citizen-participation practices, increased information disclosure and transparency, enhanced SAI performance and effectiveness, and, ultimately, more inclusive societies. Data collection is very important in this process to help SAIs and CSOs build a body of empirical evidence from each initiative to better map direct outcomes rather than rely on theoretical and normative assumptions of change processes.

The following are three categories of outcomes that SAIs can aim for:

In Argentina, the president of the General Audit Office (the collegiate board that governs the SAI) is appointed by the main opposition party. In 2012, some political circles tried to remove the president on the basis of a legal technicality. The case gained wide media coverage and mobilized civic groups and opinion leaders, who publicly stood in support of the auditor general. Important to note is that citizen-engagement mechanisms promoted by the AG in his mandate had contributed to his building a strong reputation for his commitment to human rights, transparency, and citizen engagement, and CSOs became key allies to back him in his attempt to resist removal. Ultimately, because of the widespread outrage the measure generated, the president remained in office.

Philippines: The use of citizen audit approaches produced information about program implementation that, according to the Commission of Audit (COA), would have been quite challenging to access and gather, given its existing capacities and array of audit tools. In the solid waste management program (SWMP) pilot audit, familiarity of citizen auditors with the target communities and people’s waste management practices allowed for deeper inquiries into such practices and related issues (which often went beyond the questions listed in the survey instrument). Those inquiries shed light on valuable data, such as the garbage collectors’ health concerns, their complaints with regard to earning only minimum wage, and the presence of local cleanliness campaigns and garbage collection systems in some communities, which are autonomous from the city government’s waste management initiatives.

South Korea: A civic group requested that the Board of Audit and Inspection (BAI) of Korea (the Supreme Audit Institution of Korea) conduct an audit against the Masan City government for giving inappropriate benefits to the contractor of the city’s public water reclamation project. The BAI found that an excessive reimbursement was made to the contractor and recommended that the city government seek compensation for the losses and discipline the responsible officials for their mismanagement.

Apart from direct effects that are measurable in the short and medium term, the implementation of citizen-participation policies can produce indirect effects, mainly in the medium and long term, across sectors.

The types of reforms implemented by SAIs often take place within larger reform processes. However, they contribute significantly to those reform processes.

Even though determining causality between the effects and participation mechanisms implemented is a difficult task and susceptible to errors, identifying the scope where change can be produced is relevant to take note of progress and evaluate how the implemented practices contributed.

Indirect Outcomes

Public awareness of the role and mission of the SAI

The level of general public awareness of the institutional role and acting mission of SAIs—with few exceptions—is typically not very high but rather lower than desired, considering the relevance of those institutions.

That lack of awareness is one of the main deficiencies that implementation of citizen-participation and social-accountability practices by SAIs aim to correct. Such work by SAIs implies creating and linking strategies with the public to raise awareness in society and CSOs about the role of SAIs and their relevance within the institutional system of a country.

Public debate on audit results

Auditing results are not often inputs used in social debates about public policies and services that have been analysed by SAIs. Nonetheless, they gather highly important information that could support those debates if society’s desire for that knowledge were greater.

The implementation of linking practices between SAIs and the public can result in significant improvements. Incorporating audit findings in public debate on topics that have been investigated can infuse those debates with decisive information for making sound public decisions.

Independence and autonomy of the SAI

As previously mentioned, SAIs that manage to implement a successful external outreach strategy also significantly increase their recognition and legitimacy. Establishing stronger SAIs in that area allows reducing some constraints that can affect their autonomous functioning and can therefore contribute to higher levels of independence.

A greater degree of SAIs’ independence and autonomy is an indirect outcome that can result from implementing citizen participation and social accountability practices by these control agencies. At the same time, these increases in levels of independence and autonomy translate into evident improvements in the quality and effectiveness of oversight responsibilities possessed by SAIs.

Public demand for accountability

One of the main challenges that SAIs face is collaborating in the construction of a culture of accountability by public institutions, with regard to specialized state agencies and society in general. The implementation of citizen participation and social accountability within the SAIs not only can bring positive internal results in the institution, but they also have the potential to extend to the rest of the public institutions of a country, increasing the social demands for public accountability.

Based on the functions and institutional nature of SAIs, they have excellent conditions to take on the challenge of being at the forefront in constructing a public accountability culture that involves strong social demand for these types of practices for all government institutions.

Government responsiveness

The implementation of citizen participation and social accountability practices within the SAIs can produce significant improvements at all levels of government responsibility. Those improvements result from multiple factors that are analysed in this module, such as the following:

  • The empowerment of various civil society agencies based on participation in cooperation opportunities with SAIs and resulting in more citizen demands facing public bodies.
  • The increase in public legitimacy levels of SAIs based on connections constructed with the public results in an increase in social costs that certain government bodies must face when disregarding the observations and recommendations made by SAIs.

Effectiveness, efficiency, and transparency in the use of public resources; reduction and prevention of corruption

As a consequence of the results mentioned, SAIs that build significant connections with the public are in better condition to strengthen public financial management, procure improvements in the way public spending— and all state resources in general—is administered, detecting and preventing possible misallocation by public bodies and servants, and tackling corruption.

In short, SAIs that implement successful citizen participation and social accountability strategies are capable of improving the effectiveness of their public oversight and, with it, enhance the levels of transparency and efficiency in the administration of public resources.

Outcomes of SAIs’
engagement with citizens and other actors
Positive Negative
Responsive and accountable government
  • Enhanced government responsiveness and accountability
  • Greater access to government services and resources
  • Greater realization of rights
    		•
  • Refusal of accountability demands
  • Denial of government services and resources
  • Social, economic, or political exclusion
    		•
    Effectiveness of SAIs
  • Autonomous, effective, and accountable SAIs
  • Informed and comprehensive audits
  • Control of public expenditure and performance
  • Increased public awareness about the oversight system
  • Publicity of audit information and results
  • High impact of audit reports and recommendations
    		•
  • Weak and isolated SAIs
  • Audits’ dependency on government information
  • Low control of public expenditure and performance
  • Opacity of the oversight system
  • Lack of public knowledge about audits
  • Ineffective audit reports and recommendations
    		•
    Accountability system
  • Trust in government and accountability institutions
  • Strong and collaborative oversight system
    		•
  • Reduced citizen trust in oversight system
  • Ineffective accountability system
    		•
    Practices of citizen participation
  • Increased capabilities for collective action
  • Innovative forms of collaboration between government agencies with citizens and other actors
  • Deepening of networks and solidarities
    		•
  • New capacities used to co-opt CSOs and other actors
  • New linkages used to hijack SAIs for groups’ agenda
  • Tokenistic or “captured” forms of engagement
  • Lack of accountability and representation in networks
    		•
    Construction of citizenship
  • Increased civic and political knowledge
  • Greater sense of empowerment and agency
    		•
  • Increased dependence
  • Disempowerment and reduced sense of agency
    		•
    Inclusive and cohesive societies
  • Inclusion of new actors and social groups in government activities
  • Greater social cohesion across groups
  • Improved governance
    		•
  • Reinforcement of social hierarchies and exclusion
  • Increased conflict
  • Poor governance
    		•

    Any comments? Please notify us here.

    Go to Introduction
    Go to next

    Bibliography

    TPA Initiative – ACIJ (2013): “Audit Institutions in Latin America. Transparency, Citizen Participation and Accountability Indicators”.

    OECD (2013): “How can Supreme Audit Institutions Engage External Stakeholders to Enhance Good Governance?”, Concept Note - OECD with Supreme Audit Institutions of Brazil, Chile, South Africa & IDI.

    Johnsøn, J. & Søreide, T. (2013): “Methods for learning what works and why in anti-corruption: An introduction to evaluation methods for practitioners”, U4 Issue N°8, Bergen, U4 Anti-Corruption Resource Centre- Chr. Michelsen Institute.

    Gertler, P. J. et al. (2011): “Impact Evaluation in Practice”, Washington, DC: World Bank.

    Hevia, F. & Vergara-Lope Samana (2011): “How to measure participation”, CIESAS – INDESOL. “Practical Guide For Evaluating Participatory Processes”; Marc Pares, Leonardo Díaz (IGOP/UAB) & Melissa Pomeroy (OIDP/OLDP)

    “Supreme Audit Institutions and Stakeholder Engagement Practices. A Stocktaking Report”, Effective Institutions Platform, September 2014.

    “Working with Supreme Audit Institutions", Department for International Development (DFID, 2005).

    Open Budget Survey 2012, International Budget Partnership (IBP)