THE DATA CONTRACT THAT SAVED MY PIPELINES: ENFORCING THE OPEN DATA CONTRACT STANDARD FOR RELIABLE CROSS-TEAM DATA PIPELINES

Abstract

This study discusses the failure mode where the pipeline's successful completion and no meaningful output is produced due to schema modifications at the upstream that are not poised to be detected by a downstream system. The structural issue is that it is architecturally flawed in decentralized data platforms more generally, the agreement between a producing team with its consumers teams who consume that data is encoded in a place where a machine can verify. The paper suggests integrating that rather informal contract by a formal, versioned, machine-readable document, and considers the Open Data Contract Standard (ODCS), the YAML document standard under the management of the Bitol Project of the Linux Foundation, as the means. It is a typical ODCS v3 contract for a daily orders dataset that is designed and annotated that tends to combine schema, data quality rules such as domain business logic, and service-level properties and ownership into a single document to be reviewed. The following two-gate enforcement architecture is then discussed: structural gate running in continuous integration at the pull-request time, and behavioral gate running on the same data between transformation and publication that is materialized. Experience with adoption of production changes suggests that an incidence of breaking-change moves to producer build-time and that dataset onboarding effort declines, and that quality responsibility moves to the producing realm. Boundary situations where contract overhead cannot be justified are determined and a minimum adoption way has been suggested.