In a multi-year engagement, researchers from Evidence for Policy Design advised some of India’s lead governmental auditors on advanced methods in data collection and analysis to use in their work. This case study introduces those methods and highlights how establishing data flows between branches of government can improve the rigor of audits and increase transparency.
Imagine that you are charged with a very big, very important task: evaluate the effectiveness of India’s largest antipoverty programme for the rural poor. Taxpayers want to know if their money is being spent effectively, and it’s a lot of money – roughly 3.39 trillion rupees ($50.5 billion) over the last decade. How would you do it?
In an ideal world, you’d gather together all the relevant data on programme implementation across all 600-plus districts in India’s 34 states and Union Territories, crunch the numbers, and see how they look. But this is not an ideal world. Your time and resources are limited, so you have to choose a subset of districts where implementation of the programme will reflect that across the country as a whole. How do you select the districts to audit?
You know that the only way to avoid bias-related errors in such an evaluation is to choose the districts randomly. So you decide to take all those districts, put them in a hat (so to speak) and draw out however many you can afford to audit. Then you audit them, multiply your results to reflect the whole country, and that should produce a fair assessment.
If you took this approach, you’d be doing pretty well, according to the guidelines of the Comptroller and Auditor General, or CAG, the body within the Government of India charged with actually conducting such performance audits. The CAG used this method to select districts to audit during a countrywide evaluation of the Mahatma Gandhi National Rural Employment Guarantee Act (MGNREGA) workfare programme in 2013.
But recall that this is a programme for the rural poor. You took all of India’s districts and threw them in the hat, and a large percentage were too urban or too wealthy to contain many recipients. Wasn't auditing those districts a waste of resources?
Now add this: if you have a quick look at data routinely collected in the programme’s database, you see certain red flags. Beneficiaries of a workfare programme must log in “person days” of work to receive payment, and in a few districts, programme expenditures far exceed the number of recorded person days. It looks like fraud, but it escaped your analysis since it only happened in a few districts, and they stayed in the hat when you made your random draw.
This question – how best to approach an audit of MGNREGA – was exactly the one that senior officials at the CAG posed to researchers at Evidence for Policy Design (EPoD) in 2014. The two organizations were attempting to find common areas of focus for a possible multi-year engagement, so the CAG gave the Harvard-based group an audit they had already conducted and asked how they would have done it differently. EPoD produced an Evidence Brief detailing how, if the auditors had used an alternative sampling method, they would have been able to pay special attention to both the districts with a large eligible population and the “red flag” districts, and still preserve the random nature and geographical breadth of the sample. We describe the method below.
EPoD continued its collaboration with auditors from the CAG over the following two years. As part of the Building Capacity to Use Research Evidence (BCURE) programme, funded by UK Aid, the researchers provided technical assistance on two live audits in 2015. They also conducted trainings of the CAG’s Indian Audits and Accounts Service officers, assisted in writing survey questionnaires, and trained field surveyors in new, tablet-based methods.
In this case study, we will track the first steps in the collaboration and introduce the sampling methods and other changes in practice that EPoD recommended to the CAG. Advanced approaches to data analysis may seem obtuse and beyond the scope of most policy actors' day-to-day needs, yet they can empower direct improvements to the efficiency of government processes. In rich countries, governments have the resources to disseminate new and improved methods to their staff, and citizens reap the rewards. Collaborating with policy actors in developing countries to demonstrate the value of such methods is the essence of this pilot project, and of the BCURE agenda as a whole.
The office of the Comptroller and Auditor General is a large body within the Government of India with an important mandate. Employing 43,000 people across the country, it serves as the internal watchdog, able to audit any programme the government spends money on, from state-level construction projects to the world’s largest workfare programme as described above.
In a January 2016 interview, Ganga Kapavarapu, then Additional Deputy Comptroller and Auditor General, explained what an audit entails. “It’s a huge exercise of resource mobilization, a huge engagement with the entity that is carrying out the policy” – which she calls the executive – “typically a yearlong exercise.” The CAG’s reports are tabled in Indian legislatures at the Centre and the States, after which they become available to the public. They tend to get a good deal of attention within government and in the media, not least because the CAG has uncovered some of India’s most famous scams, such as the 2G spectrum scandal of 2008, which involved government officials steeply discounting telecoms licenses in return for bribes. But even in its lower-profile audits, the CAG has a wide reach and an important voice in Indian policy.
In their position, CAG auditors must continually show value. With several large-scale audits going at once, waiting for research results is not an option. So when senior officials began exploring the possibility of a collaboration with EPoD in 2013, much of what the Harvard-based researchers did seemed outside the CAG’s use. “You look for perfection, we look for practicality,” Meenakshi Sharma, then Director General of the Professional Practices Group, said in the interview. “If we keep looking for the perfect solution, by the time it comes, the need has passed and we have lost the window of opportunity.”
However, through focused, collaborative exploration, it soon became apparent that there were research methods employed by the EPoD team that could inform and benefit the CAG’s practice.
The agency conducts financial audits (reviewing the ledgers of governmental organizations and programmes), compliance audits (checking to see the extent to which policy, rules and regulations are being followed), and a third type: performance audits. According to the guidelines,
“Performance audit evaluates the efficiency, economy and effectiveness of a given entity or programme, in an effort to move beyond financial compliance and cope with the intricate questions of measuring the performance and results of [India’s] huge investments and expenditure.”
EPoD researchers and CAG auditors agreed that the performance audit showed the greatest possibility for crossover. For EPoD, it was the type that most resembled impact evaluations they and other research organizations conduct. And for the CAG’s leadership, performance audits had long presented an opportunity to go a step further than their standard function of catching inefficiency and malfeasance, and provide the government with finer analyses to guide improvements.
The first step of the collaboration was for EPoD to review a completed performance audit and propose changes in methodology. As mentioned above, EPoD recommended revamping the sampling method, but let’s have a closer look at just what that means.
The CAG’s guidelines required that the selected sample be representative of the entire population, unbiased, and of sufficient size to extrapolate results to all of India – all solid requirements. To accomplish this, auditors randomly selected districts, ensuring that each district in India was equally likely to be selected for audit.
EPoD suggested using stratified sampling. In this method, auditors would create groups (or strata) with no crossover, and then randomly sample from within those groups, thus fulfilling the random requirement. EPoD suggested basing the stratification on any or all of three indicators readily available in administrative data: rural population, MGNREGA expenditure, and person days worked.
If auditors stratified based on rural population, for example, they might divide all districts into four groups of increasing numbers of rural citizens and sample most heavily from the quarter with the greatest number. That would ensure that the larger the targeted population in a district, the higher the probability of audit. In the scenario EPoD constructed, districts in the highest quarter would face a 0.56 probability of being audited, those in the lowest, a 0.06 probability.
This would give auditors a much better view of how the programme worked in the areas that needed it the most. For example, as you can see on the maps above, the state of Andhra Pradesh had a large rural population and high level of MGNREGA spending. Clearly, this was a state where programme implementation mattered to many people’s lives. In the actual 2013 performance audit, only four of Andhra Pradesh’s 22 districts were selected for review. Stratification would have resulted in between 12 and 16 districts being audited.
To extrapolate back to the entire population, auditors would then use post-stratification weights – meaning, that those few districts audited from that lowest-rural-population quarter would be given greater weight in the calculations. This would allow the final audit to speak about the population as a whole, while being able to examine the rural population in greater detail.
Stratified sampling is a flexible mechanism. If, for example, auditors wanted to stratify based on red flags – those districts that showed irregularities in spending patterns versus days worked – they could stratify districts based on the gap between those numbers, audit 100% of districts in the most irregular stratum, then reweight accordingly. These methods are not revolutionary: in fact, the CAG already used stratified sampling in some audits. But they had usually only stratified on financial parameters (i.e. spending). EPoD’s two insights were to stratify along multiple parameters, and to select those parameters based on administrative data that could capture wider aspects of programme effectiveness.
In the next step of the collaboration, EPoD consulted the CAG on live audits in order to test methods for wider use. The first of these two focused on Social Security programmes in the state of Kerala, implemented at the lowest administrative levels, down to the village, or panchayat, governments. The goal was to analyse the effectiveness of a statewide system that gave pensions to groups in need, including senior citizens, widows, and disabled citizens – around 2.8 million recipients, according to the state’s Auditor General.
Support from UK Aid under the BCURE initiative allowed EPoD to send Harvard PhD students and EPoD India staff to Kerala. “In our normal way of doing things,” Sharma explained, “the audit objective would be quite wide: Did it or didn’t it work?” The EPoD team observed that if the auditors narrowed their initial questions, they would be able to conduct a more precise audit that could lead more directly to recommendations for program improvement.
How to determine the best questions to ask? The EPoD team consulted existing data sources. Comparing administrative data from the Social Security programme with census data revealed discrepancies in potentially eligible populations (e.g. those above a certain age or below a certain income) and populations receiving the pensions. So, rather than, “Did it or didn’t it work?” the researchers recommended asking three much more specific questions: “Who was receiving pensions that shouldn’t be?” (thus identifying inclusion errors); “Who wasn’t receiving pensions that should be?” (identifying exclusion errors); and “Were payments made on time?” The auditors could then use techniques such as oversampling of certain strata and more detailed surveys to uncover the reasons behind the inclusion and exclusion errors and delays. It was as if the auditors were archaeologists, and choosing a smaller plot to excavate would allow them to dig much deeper.
Over the summer of 2015, EPoD staff provided inputs as auditors designed the questionnaire and implemented surveys conducted with electronic tablets – a first for the local CAG office. With public programmes in India that are delivered at the lowest levels, there is always the possibility of misuse for personal or political motives. The village councillor charged with distributing funds might hold onto them until election time, then bestow them on beneficiaries as if they were gifts. The more detailed questionnaire could capture these temporal nuances (and compare this to programmatic data to speak to its quality), and also capture whether intended recipients were even aware that they were entitled to a pension.
“We decided to use evidence even at the initial stage, even before we hit the ground,” Kapavarapu explained. Zeroing in on inclusion and exclusion errors as indicators for how the pensions programme was managed amounts to hypothesis-creation. If the auditor’s purpose is only to spotlight gaps in implementation then hypotheses are irrelevant, but if her purpose is to make targeted recommendations for improvement, they are essential.
The Kerala Social Security programme is an example of the lowest-level programme the CAG audits. The next project that EPoD assisted the CAG with was at the opposite end of the spectrum: an audit of a ministry operating across states.
To look at a large ministry and ask, "Did it or didn’t it work?" would tell the government little. One way to break down this massive undertaking is to choose certain metrics and look for inconsistencies between geographical areas. As with gaps in the data on MGNREGA, using existing data would let auditors stratify the sample and look more closely at the worst- (and perhaps best-) performing areas. This requires using data from a third party at an early stage in the audit.
CAG auditors were accustomed to using evidence to set their audit objectives, but it usually came from policy papers, administrative reports, financial statements and other records maintained by the ministry itself. The problem with depending on only the executive’s records is that it risked compounding its errors rather than catching them – in effect going with the flow created by the entity being audited and possibly following in their ruts. If the ministry were erroneously spending money on an unneeded programme to the exclusion of a better one, then the CAG would never catch it because its inquiry would follow the direction of the spending.
As part of the BCURE-funded pilot, the EPoD team analysed large-scale surveys to gain a more complete perspective of how the programme was functioning, in addition to conducting an extensive literature review to collate existing knowledge in order to identify the most informative indicators, and the highest priority areas, to measure in the audit.
“We want to examine why certain people are excluded," Sharma said in the interview. "Exclusion would be very difficult to understand if we just followed what the executive was doing.” Kapavarapu picked up the thought, adding that the executive's database only tracks recipients: “They do not have the universe. So if I want to know if the executive has excluded people, and I go by their database, I will never know it.”
Charity Troyer Moore, Director of EPoD India, says this was one of the most important collaborations between EPoD and the CAG. “To my knowledge,” she says, “the CAG has never interviewed eligible non-beneficiaries in audit surveys, so getting them to do this was a major step forward.”
A central goal of the BCURE pilot projects is to present policymakers with examples of how using evidence can help them do their jobs more effectively. In this instance, the BCURE Harvard team had the luxury of working with auditors – evidence experts in their own right. With the value of evidence widely acknowledged, the researchers were freed to work alongside senior auditors to identify where existing methods could be refined to better enable performance audits to be a useful tool for policy improvement.
In the course of the collaboration, another goal emerged: to demonstrate how greater cooperation and coordination between governmental bodies would improve outcomes. Recall that some of the methods EPoD researchers recommended were already in use (though in a limited way) in some audits, and that all of the data used to identify questions and narrow objectives was readily available from other branches of government. The elements of best practice were there, but they were separated by department walls. If this is the situation broadly in developing countries, then building channels of communication between government actors is as important as building up their skills.
Ultimately, EPoD served as a catalyst in a movement toward rigor and the use of data analytics that Kapavarapu, Sharma and others were promoting at the CAG. EPoD’s role started when the movement was already afoot, but the researchers provided several essential inputs and, as Moore puts it, “stirred the pot.” First, the collaboration on the audits detailed above created a forum for the discussion of improved data collection and analysis at the highest levels of the CAG. Second, EPoD conducted three training sessions with CAG auditors. The first was based on BCURE training modules, which are designed to impart an appreciation of the value of evidence to policymakers across ministries, and the latter two were entirely tailored to CAG purposes. These generated further discussion among mid-level auditors. Third, at the close of the pilot project, EPoD produced a methodology report with a list of specific recommendations that had emerged from the project and that would promote a culture of learning, testing, and iterating at the CAG.
Kapavarapu and Sharma have both moved on to other governmental posts, but the movement they helped foster at the CAG has since born fruit. On 6 September 2016, the Comptroller and Auditor General of India Shashi Kant Sharma inaugurated a new data analytics centre that would “play a catalytic role to synthesise and integrate relevant data into auditing process.” Starting in-house, the Centre for Data Management and Analytics (CDMA) will eventually move to its own building where it will gather and analyze data for the entire organization. It promises to bring a new level of rigor to India’s governmental audits, and thereby a new level of accountability to India’s public sector. ■
Text by V. McIntyre. Design by Angela Ambroz.