In 2003, an undergraduate student at the College of William and Mary set out to write his honors thesis on the efforts of international development organizations to help communities adapt to and mitigate the effects of global climate change. His research quickly hit a roadblock: existing public data on foreign aid flows was not precise enough to enable his analysis. From that challenge, the idea for AidData.org was born: build the world’s most comprehensive and granular database of aid information, enabling better development research.
Today, AidData.org is recognized far beyond the academic realm as a leading information and data source on development programming, including comprehensive information on aid projects, geocoded data, cutting edge working papers and research briefs, and an influential blog. Bridging the gap between academia and the policy community required us to rethink how to make our data accessible and usable for users’ diverse needs.
The launch event during Global Transparency Week was attended by more than 125 participants. It featured Dr. Homi Kharas of the Brookings Institution and Dr. Alex Deghan, then-Director of the USAID Office of Science and Technology. From 2013-2014, AidData’s internal data management tools were developed and integrated with the external web portal -- and from 2014-2016, AidData.org gained 200,000 unique users.
NoSQL databases and querying of big data (MongoDB and ElasticSearch) to manage a larger, broader database
Customized and flexible workflow management for creating new coding and data management workflows, enabling data from multiple sources
Leveraging Esri GIS tools, together with Esri open source projects, for fast mapping and geocoding in the GIS map tool
A modular, API-driven approach to separating data management, data publication, data querying, and data visualization functions
The lessons learned in deploying new technologies for web GIS mapping, big data management, and workflow flexibility to build AidData 3.0 have been applied by the ACDP partners to build a wide range of data visualization and analysis tools. For example, DG used lessons learned from AidData.org’s GIS module in the creation of the MONITOR GIS tool built for USAID/Colombia, and used a similar MongoDB database design in archiving more than 100,000 Development Experience Clearinghouse (DEC) documents for USAID’s Global Development Lab through HESN.
DG has also leveraged AidData 3.0 innovations to power a wide range of data-driven development projects, such as using the NoSQL database tool to improve performance of web portals (including the Open Contracting Explorer and Corruption Risk Dashboard in Vietnam), implementing Drupal to power the Extractive Industries Transparency Initiative web portal, and leveraging advancements in geospatial data visualization in the AidData 3.0 GIS to develop the Aid Management Platform GIS portal and more than a dozen additional GIS portals (including that of the Complex Emergencies and Political Stability in Asia (CEPSA) program, built in partnership between ACDP members University of Texas at Austin and DG).
Figure 1: CEPSA Nepal GIS Map
How AidData.org is Used
For the first time, AidData.org’s dashboards and GIS modules provided users with the ability to analyze and explore AidData’s 1 million-plus records representing $40 trillion in development financing in real time without the use of statistical software. This made its data accessible to practitioners and policy-makers without the time or capacity to download and analyze raw data.
USAID Staff: AidData.org’s GIS features are frequently used by USAID Geocenter team members to analyze and download geocoded data as they prepare for global visits to USAID Missions:
"The data collected and published by AidData is a great starting point for when I work with a mission that lacks GIS capacity. Checking the AidData website is one of the first things I do when I receive a request from a country I haven't worked in before. The ability to immediately show how mapping development interventions can provide a highly valuable visual resource for decision-making makes GIS integration a no-brainer."
USAID GeoCenter Team Member
AidData Staff: AidData.org staff uses the internal tools to gather, clean, geocode, and publish data to the AidData.org web portal. By combining all of these functions in one suite capable of managing millions of records, time and effort required for data management has been significantly reduced. Further, the AidData.org internal data management tools were used to generate the 2016 launch of the 3.0 research release, and have minimized the time required to download and republish the latest OECD data on development activities.
"The AidData internal software is a critical, catalyzing tool that has enabled efficient, accurate data management across the organization and beyond. It is a tool that has saved time for regular data management tasks such as importing new sources of data (such as the OECD CRS data, annual report data for donors outside the DAC, etc). It has also enabled value-add work that would have been effectively impossible without it, including performing double-blind and arbitration coding to assign sub-national locations to development finance activities and to assign activity codes to capture the full range of activities within each development project. The internal software will provide a solid foundation for AidData's work now and well into the future."
Brooke Russell (AidData Assistant Data Manager)
Researchers: Researchers use the web portal to access the research release, download geocoded datasets, and disseminate their research through the web portal and blog. Since 2013, more than 250 scholarly articles have cited the AidData.org data in their publication.
Other Users: AidData.org has been accessed more than 225,000 times since its launch in 2013, with tens of thousands of data and research downloads used for various purposes.