I have spent years designing and operating data pipelines in Google Cloud, and one thing has not changed: resilience is not optional. It does not matter how nice your design diagrams look or how scalable the architecture is. In practice, nodes die, quotas are exhausted, regions are shaded, schemas alter unannounced, and message queues are clogged up at the most unpredictable moments. The main distinction between a functional pipeline and a resilient pipeline lies in the fact that the former can withstand failures and still meet latency requirements.

The article explains my philosophy on resilience in distributed data pipelines on GCP, based not only on the experience of running these systems, but also more broadly on systems research and Google operational experience.