I am the lead at the Data Automation Lab in the Development Economics Debt Statistics group at the World Bank. We're initiating the movement away from manual data processing, freeing up time for increasing data quality and conducting more analysis.
Our first program's process map.
The Lab serves to automate the data preparation process for the International Debt Statistics
(IDS) database and annual publication. IDS provides a wealth of statistics and analysis on the external debt stocks and financial flows (debt and equity) of the world’s economies in 2017. It offers more than 200 time-series indicators from 1970 to 2017 for most reporting countries.
The Debt Statistics team manages a database of over 250,000 individual debt instruments. Every year, countries and multilaterals send us their new financial obligations. The data is reported in various formats, leading to months of manual data processing.
Automation offered a unique opportunity to increase efficiency and standardization in data processing. Less time and labor is now spent on repetitive tasks, such as cleaning, coding, and formatting data. Methods of preparation are also becoming codified. Accordingly, the Debt Statistics team created a “Data Automation Lab.” I serve as the lead, focusing on developing reusable and modular Python code that automates these repetitive tasks.
A department-wide presentation on our first results.