Researchers rely on metadata systems to navigate, interpret, and prepare complex survey data for analysis. Traditional metadata in the social sciences are formatted as questionnaires, codebooks, and other written documentation, which require a “massive professional investment” to use proficiently. As machine-learning methods become more popular, researchers need to work with hundreds or thousands of variables, but existing document-based metadata systems do not scale well to high-dimensional data. The Fragile Families Challenge (FFC), a mass collaboration applying the common task framework to predict six outcomes using Fragile Families and Child Wellbeing Study (FFCWS) data, revealed that data preparation was a major obstacle. Participants reported difficulty extracting basic variable information, requested machine-readable metadata, and sometimes constructed metadata fields independently. This motivated the authors to redesign the FFCWS metadata system based on the principle of “treating metadata as data.”
The redesign was informed by observing FFC participants through six “getting started” workshops (each three hours), weekly virtual office hours, an FFC email address, six informational interviews with dedicated participants, review of code from many submissions, and a two-day workshop where participants provided feedback on prototypes. The authors undertook four tasks: (1) standardizing canonical FFCWS data files (variable names and missing data codes); (2) producing new machine-actionable metadata (variable types, warning flags, grouping similar questions, variable topics/subtopics, focal person, scales/measures, and question text); (3) integrating automated tests into the metadata build process; and (4) creating tools for metadata access (a Web API, R and Python packages, and a Web application). The metadata were stored in a single CSV file. Variable names were standardized so users could infer wave, respondent, source (questionnaire vs. constructed), and question location. Missing data codes were standardized to match core variables. Variable types were classified as continuous, unordered categorical, ordered categorical, binary, or string. Six types of warning flags were created (e.g., misordered response options). Similar/identical questions across waves were grouped using a partially automated process based on variable labels and questionnaire text. Topic tags were hierarchically organized (e.g., “demographics” with subtopics like “age” and “race/ethnicity”). Focal person (child, father, mother, primary caregiver, partner, other) was identified for each variable. A scale field was added for widely used scales and measures. Full question text was programmatically extracted and edited. Automated tests checked for correct value ranges and impossible combinations (e.g., no teacher respondent in wave 1).
The FFCWS data set contains approximately 900 variables at baseline and more than 3,200 variables by the age 9 follow-up. The challenge file contained 12,942 variables. The redesigned metadata system includes a single CSV file with comprehensive fields. The authors conducted three informal evaluations: (1) pilot versions were presented at the FFC workshop, and feedback led to redesign of several features; (2) a comparison of specific tasks (Table 4) showed that many tasks are substantially easier with the new metadata; (3) the FFC was run as an assignment in an undergraduate machine learning class in spring 2017 (old metadata) and spring 2018 (new metadata). Average predictive performance was similar across years, but students reported substantially fewer complaints with the new metadata infrastructure. The authors note that these evaluations are not definitive but give confidence that the new system is an improvement.
**Clinical Implications**
While this paper is not a clinical study, its findings have implications for medical and public health research that relies on complex longitudinal surveys. Improved metadata infrastructure can reduce errors and time spent on data preparation, enabling researchers to focus on substantive questions. The open-sourced tools can be adapted for other large-scale surveys, potentially improving the efficiency and reproducibility of research in fields such as epidemiology, health services research, and social determinants of health. The authors recommend that data creators provide standard, machine-actionable metadata fields (especially variable type and substantive topic) and release metadata in machine-readable formats like CSV to facilitate the use of machine-learning methods.