Skippr Docs

Appearance

CDC Destinations ​

When CDC is enabled, Skippr applies changes to the destination using idempotent MERGE operations that guarantee exactly-once final state. Every destination gets two automatic additions:

Order token column ​

Skippr adds a _skippr_order_token column (type VARCHAR / STRING) to every CDC-managed table. This column stores a lexicographically sortable token derived from the source's native log position (e.g. PostgreSQL LSN, MySQL binlog position).

During upsert, a row is only written if the incoming order token is greater than the existing token. This rejects stale writes and duplicate replays.

Tombstone table ​

For each CDC-managed table, Skippr creates a companion tombstone table named _skippr_tombstones_{table}. When a delete is applied, the business key and order token are recorded in the tombstone table.

A subsequent insert for a deleted key is blocked unless its order token proves it occurred after the delete. This prevents "ghost resurrections" from replayed or out-of-order insert events.

Snowflake ​

Skippr applies CDC changes to Snowflake using MERGE DML statements.

  • Upsert: MERGE INTO target USING staging ON business_key WHEN MATCHED AND staging._skippr_order_token > target._skippr_order_token THEN UPDATE ... WHEN NOT MATCHED THEN INSERT ...
  • Delete: Inserts a tombstone record and deletes the target row in a single MERGE transaction
  • DDL: _skippr_order_token VARCHAR column and _skippr_tombstones_{table} table are created automatically

No additional Snowflake configuration is required beyond the standard Snowflake destination setup.

BigQuery ​

Skippr applies CDC changes to BigQuery using MERGE DML statements with the BigQuery SQL dialect.

  • Upsert: MERGE with order-token comparison in the WHEN MATCHED clause
  • Delete: Tombstone insert + row deletion in a single MERGE statement
  • DDL: _skippr_order_token STRING column and tombstone table created automatically

BigQuery MERGE statements are atomic and consistent, providing exactly-once semantics.

PostgreSQL ​

Skippr applies CDC changes to PostgreSQL using a staging table + INSERT ... ON CONFLICT pattern.

  • Upsert: Data is loaded into a staging table, then merged into the target with INSERT ... ON CONFLICT (business_key) DO UPDATE SET ... WHERE staging._skippr_order_token > target._skippr_order_token
  • Delete: Tombstone insert + DELETE FROM target WHERE EXISTS (tombstone match)
  • DDL: _skippr_order_token TEXT column and tombstone table created automatically

The staging-then-merge pattern allows bulk loading while maintaining order-token guards.

Redshift ​

Skippr applies CDC changes to Redshift using a staging table + MERGE pattern.

  • Upsert: Data is COPY-loaded into a staging table, then merged with MERGE INTO target USING staging with order-token guards
  • Delete: Tombstone insert + conditional delete
  • DDL: _skippr_order_token VARCHAR(MAX) column and tombstone table created automatically

Redshift's MERGE support (available since late 2023) provides atomic upsert semantics.

ClickHouse ​

Skippr applies CDC changes to ClickHouse using ReplacingMergeTree engine semantics.

  • Upsert: Rows are inserted with order tokens; ClickHouse's ReplacingMergeTree deduplicates by keeping the row with the highest version/order token during merges
  • Delete: ALTER TABLE ... DELETE WHERE with tombstone tracking
  • DDL: _skippr_order_token String column and tombstone table (standard MergeTree) created automatically

ClickHouse's eventual-consistency merge process means recent duplicates may be visible until a background merge occurs. Use FINAL in queries for point-in-time consistency.

Databricks ​

Skippr applies CDC changes to Databricks using Unity Catalog MERGE on Delta tables.

  • Upsert: MERGE INTO target USING staging ON business_key with order-token comparison
  • Delete: Tombstone insert + conditional delete via MERGE
  • DDL: _skippr_order_token STRING column and tombstone table created automatically

Delta Lake's ACID transactions ensure exactly-once semantics.

Synapse ​

Skippr applies CDC changes to Azure Synapse using MERGE statements via the Tiberius driver.

  • Upsert: MERGE target USING staging ON business_key WHEN MATCHED AND staging._skippr_order_token > target._skippr_order_token THEN UPDATE ... WHEN NOT MATCHED THEN INSERT ...
  • Delete: Tombstone insert + conditional delete
  • DDL: _skippr_order_token NVARCHAR(4000) column and tombstone table created automatically

MotherDuck ​

Skippr applies CDC changes to MotherDuck using DuckDB's MERGE (via INSERT OR REPLACE with order-token guards).

  • Upsert: INSERT OR REPLACE with a subquery that checks order-token ordering
  • Delete: Tombstone insert + DELETE with tombstone join
  • DDL: _skippr_order_token VARCHAR column and tombstone table created automatically

MotherDuck inherits DuckDB's ACID transaction guarantees.