Quick Start

Define a pipeline in YAML. Run it.

No boilerplate code, no JVM, no Zookeeper. One config file, one binary.

          
          
          
          pipeline.yaml
        
metadata:
  name: orders-to-kafka
  tenant: acme

spec:
  source:
    type: mysql
    config:
      dsn: ${MYSQL_DSN}
      tables: [shop.orders]

  processors:
    - type: javascript
      inline: |
        function processBatch(events) {
          return events.filter(e => e.after?.status !== 'draft');
        }
    - type: flatten
      separator: "__"
      empty_list: drop

  sinks:
    - type: kafka
      config:
        brokers: ${KAFKA_BROKERS}
        topic: cdc.${source.table}
        envelope:
          type: debezium

Capture

Connect to MySQL binlog or PostgreSQL logical replication. Wildcard table patterns supported.

Processors

Chain native and scripted processors in order. Here: JavaScript drops draft orders, then Flatten normalizes nested JSON to parent__child keys before delivery.

Deliver

Fan-out to Kafka, Redis, and NATS simultaneously. Each sink is independent — one failure won't block the others.

Route

Template strings resolve per-event: cdc.${source.table} routes each table to its own topic.

Envelope

Choose the wire format consumers expect: Native, Debezium-compatible, or CloudEvents 1.0. One config field, no consumer code changes.

Checkpoint

At-least-once delivery with configurable commit policies. Resume exactly where you left off after restarts.

Why DeltaForge

Built for production (CDC)

Stream database changes reliably with minimal overhead.

High Performance

Single binary, no JVM, no GC pauses. Predictable sub-millisecond latency with minimal memory footprint — handles high-throughput workloads without surprises.

Reliable Checkpoints

At-least-once delivery guaranteed. Checkpoints are saved only after sink acknowledgement — never before. Configurable commit policies: all, required, or quorum.

Initial Load & Snapshot

Consistent snapshot of existing table data before streaming begins. Tables run in parallel with chunked range reads for large datasets. Lock-free InnoDB approach — no RELOAD privilege required. Resumes at table granularity after interruption.

Dynamic Routing

Route events to per-table topics, streams, or subjects using template strings or JavaScript logic. No pipeline restart required when routing rules change.

Multi-Sink Fan-Out

Deliver concurrently to Kafka, Redis, and NATS in a single pipeline. Each sink is independently required or best-effort — one failure won't block the others.

Schema Discovery

Automatically infer and track schemas from live event payloads. Detects structural drift in nested JSON and alerts before it breaks downstream consumers.

Live Observability

Prometheus metrics, structured logging, and health probes built in. Pause, resume, patch, and inspect running pipelines via REST API — no redeployment needed.

Zero-Downtime DDL

Detects schema changes and reloads automatically. No pipeline restart, no manual intervention, no dropped events during table alterations.

Graceful Failover

Handles source failover with automatic schema revalidation. Reconnects with exponential backoff and jitter — resumes exactly where it left off.

Output Formats

Three envelope formats, one config field

Match the wire format your consumers expect. Switch with a single line of YAML.

Native Lowest overhead

Minimal envelope for DeltaForge-native consumers. No wrapper overhead.

{ "op": "c", "after": { "id": 1, "status": "new" }, "source": { "table": "orders" } }

Debezium Drop-in compatible

Schemaless mode compatible with JsonConverter schemas.enable=false.

{ "schema": null, "payload": { "op": "c", "after": { "id": 1, "status": "new" } } }

CloudEvents CNCF standard

CNCF CloudEvents 1.0 spec for event-driven and serverless architectures.

{ "specversion": "1.0", "type": "com.acme.orders.create", "data": { "after": { "id": 1, "status": "new" } } }

Dynamic Routing

Route every event to the right destination

Template strings for simple per-table routing. JavaScript for conditional business logic.

# Routes each table to its own topic automatically sinks: - type: kafka config: topic: cdc.${source.db}.${source.table} key: ${after.customer_id} - type: redis config: stream: events:${source.table} key: ${after.id} - type: nats config: subject: cdc.${source.db}.${source.table} # Available: ${source.table}, ${source.db}, ${op}, # ${after.<field>}, ${before.<field>}, ${tenant_id}

Install

Up and running in seconds

Single binary. No runtime dependencies.

Docker

Recommended

docker pull ghcr.io/vnvo/deltaforge:latest copy

Docker Hub

Alternative registry

docker pull vnvohub/deltaforge:latest copy

From Source

Requires Rust 1.89+

cargo build --release -p runner copy