We are all familiar with drifting in-app configuration and IaC. We’re starting with a specific configuration backed with IaC files. Soon after, we are facing a “drift” or a change between what is actually configured in our infrastructure and files. The same behavior happens in data. The schema starts in a specific shape. As data ingestion grows and scales to different sources, we get a schema drift, a messy, unstructured database and an analytical layer that keeps failing due to a bad schema. In this article, we will learn how to deal with the scenario and how to work with dynamic schemas.

Schemas Are a Major Struggle

A schema defines the structure of the data format. Keys/Values/Formats/Types, a combination of all, results in a defined structure or simply—schema.