Previous Articles on CockroachDB CDC

Using CockroachDB CDC with Apache Pulsar
Using CockroachDB CDC with Azure Event Hubs
Using CockroachDB CDC with Confluent Cloud Kafka and Schema Registry
SaaS Galore: Integrating CockroachDB With Confluent Kafka, Fivetran, and Snowflake
CockroachDB CDC Using Minio as Cloud Storage Sink – Part 3
CockroachDB CDC using Hadoop Ozone S3 Gateway as cloud storage sink

Motivation

Protocol Buffers are language-neutral, platform-neutral extensible mechanisms for serializing structured data. It’s a common choice for platforms needing to pass messages between systems. CockroachDB is a distributed SQL database built on a transactional and strongly-consistent key-value store. It scales horizontally; survives disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention; supports strongly consistent ACID transactions; and provides a familiar SQL API for structuring, manipulating, and querying data.

There is no official support for Protocol Buffers in CockroachDB Changefeeds, even though we use Protocol Buffers extensively in the code. A recent customer conversation led to this experiment where I’m going to use several recent features to demonstrate the ability to serialize CockroachDB rows to proto and emit via CDC Queries. This is the first time we’re looking at CDC Queries. This is a new flexible way to express CockroachDB streams.

Leave a Reply

Your email address will not be published. Required fields are marked *