Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Examples

Complete pipeline configurations demonstrating common DeltaForge use cases. Each example is ready to run with minimal modifications.

Available Examples

ExampleSourceSink(s)Key Features
MySQL to RedisMySQLRedisJavaScript processor, PII redaction
Turso to KafkaTurso/libSQLKafkaNative CDC, CloudEvents envelope
PostgreSQL to NATSPostgreSQLNATSLogical replication, CloudEvents
Multi-Sink Fan-OutMySQLKafka + Redis + NATSMultiple envelopes, selective checkpointing
Event FilteringMySQLKafkaJavaScript filtering, PII redaction
Schema SensingPostgreSQLKafkaJSON schema inference, drift detection
Production KafkaPostgreSQLKafkaSASL/SSL auth, exactly-once, tuning
Cache InvalidationMySQLRedisCDC stream for cache invalidation workers
Audit TrailPostgreSQLKafkaCompliance logging, PII redaction
Analytics PreprocessingMySQLKafka + RedisMetrics enrichment, analytics stream
Outbox PatternMySQLKafkaTransactional outbox, raw payload, per-aggregate routing

Quick Start

  1. Set environment variables for your database and sink connections
  2. Copy the example to a .yaml file
  3. Run DeltaForge: 
    cargo run -p runner -- --config your-pipeline.yaml

Examples by Category

Getting Started

ExampleDescription
MySQL to RedisSimple pipeline with JavaScript transformation
Turso to KafkaEdge database to Kafka with CloudEvents
PostgreSQL to NATSPostgreSQL logical replication to NATS

Production Patterns

ExampleDescription
Production KafkaAuthentication, exactly-once, performance tuning
Multi-Sink Fan-OutMultiple sinks with different formats
Cache InvalidationCDC stream for cache invalidation
Outbox PatternTransactional outbox with raw payload delivery

Data Processing

ExampleDescription
Event FilteringFilter, drop, and redact events
Schema SensingAutomatic JSON schema discovery
Analytics PreprocessingPrepare events for analytics platforms

Compliance & Auditing

ExampleDescription
Audit TrailSOC2/HIPAA/GDPR-compliant change tracking

Examples by Feature

Envelope Formats

FormatExample
NativeMySQL to Redis, Event Filtering, Cache Invalidation
DebeziumSchema Sensing, Multi-Sink Fan-Out, Production Kafka
CloudEventsTurso to Kafka, PostgreSQL to NATS

JavaScript Processors

Use CaseExample
PII RedactionMySQL to Redis, Audit Trail
Event FilteringEvent Filtering
EnrichmentTurso to Kafka, Analytics Preprocessing
Cache Key GenerationCache Invalidation
Audit MetadataAudit Trail

Multi-Sink Patterns

PatternExample
Fan-out with different formatsMulti-Sink Fan-Out
Primary + best-effort secondaryMulti-Sink Fan-Out, Analytics Preprocessing

Customizing Examples

Change Envelope Format

All sinks support configurable envelopes. Add to any sink config:

sinks:
  - type: kafka
    config:
      # ... other config
      envelope:
        type: cloudevents          # or: native, debezium
        type_prefix: "com.example" # required for cloudevents
      encoding: json

See Envelopes and Encodings for details.

Add Multiple Sinks

Fan out to multiple destinations with different formats:

sinks:
  - type: kafka
    config:
      id: primary-kafka
      envelope:
        type: debezium
      required: true    # Must succeed for checkpoint
  - type: redis
    config:
      id: cache-redis
      envelope:
        type: native
      required: false   # Best-effort, won't block checkpoint

See Sinks documentation for multi-sink patterns.

Enable Schema Sensing

Automatically discover JSON structure in your data:

schema_sensing:
  enabled: true
  deep_inspect:
    enabled: true
    max_depth: 3
  sampling:
    warmup_events: 100
    sample_rate: 10

See Schema Sensing for configuration options.

More Resources