A project’s theory of change lays out how the project’s desired outcomes might be achieved. It maps out which changes need to happen to reach those outcomes. However, a theory of change is based on a number of assumptions and is not always correct. Qualitative Comparative Analysis (QCA) uses qualitative and quantitative data to rigorously test theories of change, assessing the impact of key factors on project outcomes in different contexts. QCA is one way of making systematic comparisons across contexts. It takes into account various complex and dynamic factors which may affect project success in a range of contexts.
QCA doesn’t simply test a single, linear pathway for change. It tests multiple pathways to assess how combinations of factors lead to success in a given context as well as across contexts. It also complements theory-based evaluations testing the relationship between particular aspects of a project’s theory of change and their overall outcomes. This allows evaluators to test assumptions and linkages between outputs and outcomes.
Rather than relying on statistical models, QCA uses formal logic models to test relationships between key factors. The factors being assessed are rated on a scale (reflecting the extent to which they are present in a given context), and then input into a ‘truth table’ which lines them up with the outcome being assessed. The data from the table is then analysed to assess which factors, or combinations of factors are necessary or sufficient for a successful outcome.
How QCA works in practice
Coffey recently used QCA for the evaluation of the Africa Regional Empowerment and Accountability Programme, funded by the UK government. The program works in 36 countries across Africa to build civil society voice and empower citizens to hold government to account. In this case, QCA was used to compare projects implemented by three different partners across a sample of eight African countries. The results were then generalised across the whole program.
Theories of change identified key contextual and project-related factors, like the level of political stability, strength of national civil society and the strength of local partner organisations. Coffey’s evaluators then used QCA to test the relationship between these factors and success.
A ‘truth table’ for the project could look like this (all information is indicative):
In this case, QCA, when combined with other approaches, such as contribution analysis, identified stronger findings than would otherwise have been possible. It identified the key factors which are necessary for strong civil society organisations to hold government to account, specific to different types of governance contexts. It also tested the thinking behind the programme, where certain activities were found not to be necessary to achieve required outcomes.
When to use QCA
QCA is not a ‘silver bullet’ and cannot be used in all cases. It can only be used where there is a detailed theory of change and the project is implemented in different, clearly defined contexts, including by different partners or using different modalities. High quality, consistent qualitative data is needed across each context, project or modality to enable comparison. Specific technical skills and software are also required for QCA analysis.
There are five keys to success when using QCA:
1. Have a strong, detailed theory of change. QCA is theory-based, so its strength comes from the quality of the theory it is based on.
2. Have clearly defined cases. QCA is a comparative approach, so it requires clear cases to compare. These can be cases where the project is working in different contexts or through different partners.
3. Define key factors for success, based on the theory of change, which can be assessed using existing secondary data or primary data gathered through the evaluation.
4. Gather consistent qualitative or quantitative data for all key factors, for each case, so the cases can be compared against one another.
5. Set clearly defined thresholds for success, so that each factor can be assessed against the given outcome and the linkages between the QCA analysis and the theory being tested are clear.
Click here to find out more about applying Qualitative Comparative Analysis.