Lead authors: Sebastián Pilo (ACIJ)
Contributing authors: Sruti Bandyopadhyay (World Bank)
- Module 04
- Indicators to assess participatory mechanisms
Indicators to assess participatory mechanisms for SAISs
Importance: Very high
One of the main causes for failure in participation policies often lies in the weakness of communication strategies. For strategies to be successful, they must be known by all— or at least a significant percentage—of the targeted population. If only a small number of citizens and CSOs who could participate know of the existence of said possibility, not only will the opportunity to gain citizen engagement be lost but also the expected results and the contribution will be limited.
That statement does not imply that the levels of information and the contribution of those channels cannot increase over time. However, the objectives and strategies for each stage must be previously planned for its execution in reasonable phases.
As a result, those who plan and execute these policies must permanently verify the level of knowledge that certain targeted organizations or citizens have regarding the following:
- The existence of SAIs
- Their goals, mandates, missions, and characteristics
- The conditions and modality for participation
- The ways in which their contributions will be channelled
- How many citizens or organizations know of the existence of this participation mechanism?
- How many are not aware of it?
- What efforts have not yet been aimed at informing the targeted public about the existence of this mechanism and the possibility to participate in it?
- How complete is the information that potential participants have about the nature of this mechanism?
The expected level of participation may vary according to the characteristics of the various practices implemented. In some cases, mass participation may be expected; in others, the targeted audience may only include a few people, whether individually or by represented CSOs. However, in all cases, avoiding participation exclusiveness is a permanent challenge.
The active levels of participation depend on several variables, including (a) those related to the actual practice— the relevance of the practice’s subject, its methodological scheme, the ability to adequately identify a target population, the implementation of effective communication; and (b) those related to the context— level of citizen participation in the public, interest generated about the topic, legitimacy of the summoning institution, and citizen confidence about the effectiveness to incorporate their contributions to the institution, among others.
Measuring the effectiveness of those practices in terms of active participation implies comparing those levels with those that are desirable for the specific activity implemented.
Low participation levels imply losing practice potential and can also result in diminishing the legitimacy of the institution in the long run. In cases in which active participation results end up being significantly lower than desired, an important step is to follow up on the shortfall according to the outlined variables—or those applicable to the given situation—to correct for weaknesses.
On the other hand, successful levels of participation imply a greater challenge in terms of fulfilling expectations and guaranteeing sustainability over time.
- Is the level of active participation compatible with the desirable level? That is, upon developing the strategy, has the institution proposed to involve all those whose participation makes sense?
- How many of those that would be interested in this tool have effectively benefited from it?
- What are the main reasons for some of those in the participation mechanism, who seem to find relevance in such practices, not yet being fully engaged?
Importance: Very High
Beyond the active engagement levels that are reported within the participation mechanisms implemented, evaluating the levels of satisfaction that participants—active and potential—exhibit with respect to how implementation is executed is highly relevant. That process includes giving feedback on the proposed method, the way in which the participation activities are carried out, and the characteristics of the following stages— in terms of the institution’s receptiveness to citizens’ contributions, the creation of feedback, and the sustainability of the practice, among others.
Different measuring systems of opinion and feedback—such as surveys, focus groups, and personalized consulting—are necessary to obtain information related to participant satisfaction levels—including that of potential participants—with the mechanism, the points of nonconformity with regard to the mechanism, and eventual suggestions for its improvement.
- What is the general level of satisfaction that the participants—active and potential—exhibit with respect to the mechanism proposed?
- More specifically, what is the level of satisfaction that they exhibit with respect to the following aspects:
- Topics of the participation request
- Defining of the target audience
- The way in which the announcement was made
- Suitability of the mechanism, keeping in mind the modality is developed to channel their contributions
- The institution’s receptiveness and permeability in terms of the contributions made
- Sustainability of the mechanism
- What is the specific discrepancy that exists with respect to the mechanism?
- Do participants have any suggestions for improvement? How receptive is the institution to such suggestions?
If feedback and suggestions received through public participation and social accountability do not have an effect on the existing practices of SAIs, citizen-engagement practices will be greatly limited— and in some cases, even counterproductive. One of the greatest risks is that they end up being perceived as mere tokenism and have the purpose only of legitimizing previously defined decisions.
Opening channels for citizen participation, therefore, implies SAIs’ responsibility to generate effective acceptance in the auditing policies, transforming citizen contributions into valuable input to the oversight duty.
Those citizens and CSOs that participate in these instances should be able to perceive that their contributions translate into substantial improvements in public financial management and specifically in control over transparency and efficiency in the use of resources and public budgets.
Measuring the effective results of such practices is therefore not possible, but the way in which they have managed to influence the public decision makers in terms of shaping policies and actions of the oversight institution can be evaluated.
- Is the institution tolerant of the contributions made by the particular civil society and the public in general?
- Are the public servants receptive to the value of those contributions and committed to effectively considering them as input to their work?
- What are the specific changes in the functioning of the institution that have created this practice?
- How many auditing exercises have incorporated citizen contributions? How relevant were they?
- Have citizen contributions created substantial improvements in public financial management?
Those who become involved in citizen participation and social accountability instances developed by the SAIs must know the effects and results of their contributions. It is the only way to validate the commitments made by citizens, which allows them to value the mechanism and justify its continuance.
Citizens who are unaware whether or not their contributions were a valuable input to auditing processes will not be able to measure the value of their work or become motivated to continue with their participation. They will be less willing to participate next time.
As a result, giving feedback to citizens and periodically evaluating the level of compliance are fundamental responsibilities for SAIs.
- Did the actors involved in these citizen participation or social accountability instances receive feedback on their contributions?
- Do those actors know how their contributions generated change or new directions in the institution?
- Have said actors been informed about how the SAI values their contributions and considers them useful?
Finally, what is left to ask is related to the institutionalization levels of participatory mechanisms or social accountability that are implemented. Although standardizing those instances is not always necessary (especially when they are being developed), institutionalizing the actions that are effectively developed is valuable. In many cases, therefore, the norm that regulates a mechanism of this nature can be, not an action that begins a practice, but one that recognizes, institutionalizes, and organizes it hierarchically.
Consolidated practices that are not formally institutionalized are usually more vulnerable to circumstantial fluctuations, change of leadership, and tend to be less predictable.
- Has the implemented participatory or social accountability mechanism been formally standardized?
- What are the degrees of institutionalization of these practices that the institution carries out?
- Has the normative plan of the mechanism benefited from practical experience?
- Are the levels of mechanism institutionalization sufficient to guarantee the possibility for consolidation and stability through time?
2015 Copyright - World Bank Institute & ACIJ
TPA Initiative – ACIJ (2013): “Audit Institutions in Latin America. Transparency, Citizen Participation and Accountability Indicators”.
OECD (2013): “How can Supreme Audit Institutions Engage External Stakeholders to Enhance Good Governance?”, Concept Note - OECD with Supreme Audit Institutions of Brazil, Chile, South Africa & IDI.
Johnsøn, J. & Søreide, T. (2013): “Methods for learning what works and why in anti-corruption: An introduction to evaluation methods for practitioners”, U4 Issue N°8, Bergen, U4 Anti-Corruption Resource Centre- Chr. Michelsen Institute.
Gertler, P. J. et al. (2011): “Impact Evaluation in Practice”, Washington, DC: World Bank.
Hevia, F. & Vergara-Lope Samana (2011): “How to measure participation”, CIESAS – INDESOL. “Practical Guide For Evaluating Participatory Processes”; Marc Pares, Leonardo Díaz (IGOP/UAB) & Melissa Pomeroy (OIDP/OLDP)
“Supreme Audit Institutions and Stakeholder Engagement Practices. A Stocktaking Report”, Effective Institutions Platform, September 2014.
“Working with Supreme Audit Institutions", Department for International Development (DFID, 2005).
Open Budget Survey 2012, International Budget Partnership (IBP)