ballerinax/cdc Ballerina library

Overview

The Change Data Capture (CDC) connector provides APIs to capture and process database change events in real-time. It enables developers to define services that handle change capture events such as inserts, updates, and deletes. Built on top of the Debezium framework, it supports popular databases like MySQL, Microsoft SQL Server, PostgreSQL, and Oracle.

Key Features

• Capture real-time change events (create, read, update, delete) from databases
• Support for multiple database systems including MySQL, MSSQL, PostgreSQL, and Oracle
• Event-driven service model with dedicated handlers for each operation type
• Built on the Debezium framework for reliable change data capture

Quickstart

Step 1: Import Required Modules

Add the following imports to your Ballerina program:

• ballerinax/cdc: Core module that provides APIs to capture and process database change events.
• ballerinax/mysql: Provides MySQL-specific listener and types for CDC. Replace with the corresponding module for your database if needed.
• ballerinax/mysql.cdc.driver as _: Debezium-based driver for MySQL CDC. Use the appropriate driver for your database (e.g., mssql.cdc.driver, postgresql.cdc.driver, or oracledb.cdc.driver).

import ballerinax/cdc;
import ballerinax/mysql;
import ballerinax/mysql.cdc.driver as _;

Step 2: Configure the CDC Listener

Create a CDC listener for your MySQL database by specifying the connection details:

listener mysql:CdcListener mysqlListener = new ({
    database: {
        hostname: "localhost",
        port: 3306,
        username: "username",
        password: "password",
        includedDatabases: ["inventory"]
    }
});

Step 3: Define the CDC Service

Implement a cdc:Service to handle database change events:

service on mysqlListener {

    remote function onRead(record {} after) returns cdc:Error? {
        // Handle the read event
        log:printInfo(`Record read: ${after}`);
    }

    remote function onCreate(record {} after) returns cdc:Error? {
        // Handle the create event
        log:printInfo(`Record created: ${after}`);
    }

    remote function onUpdate(record {} before, record {} after) returns cdc:Error? {
        // Handle the update event
        log:printInfo(`Record updated from: ${before}, to ${after}`);
    }

    remote function onDelete(record {} before) returns cdc:Error? {
        // Handle the delete event
        log:printInfo(`Record deleted: ${before}`);
    }
}

Step 4: Run the Application

Run your Ballerina application:

bal run

Examples

The cdc module provides practical examples illustrating its usage in various real-world scenarios. Explore these examples to understand how to capture and process database change events effectively.

1. Fraud Detection - Detect suspicious transactions in a financial database and send fraud alerts via email. This example showcases how to integrate the CDC module with the Gmail connector to notify stakeholders of potential fraud.

2. Cache Management - Synchronize a Redis cache with changes in a MySQL database. It listens to changes in the products, vendors, and product_reviews tables and updates the Redis cache accordingly.

3. Connection Retries - Monitor an orders database with automatic reconnection on failures. This example configures exponential backoff retry logic so the service recovers gracefully from temporary database downtime.

4. Heartbeat with Liveness - Monitor a financial transactions table with heartbeat and liveness checking enabled. The heartbeat keeps the database connection alive during idle periods, and the liveness check ensures the CDC listener remains active.

5. Kafka Offset and Schema Storage - Synchronize inventory changes using Apache Kafka for both offset storage and schema history. This enables reliable distributed state management where the service can resume processing after a restart without missing or duplicating events.

6. S3 Schema History - Capture audit log changes with Amazon S3-backed schema history storage. This cloud-native approach stores database schema evolution in S3, removing the need for self-hosted schema history infrastructure.