How a pipeline works
A pipeline reads tables from your source and writes them into your destination, and keeps them current. This page is the map: each part links to the page that has the rules.
1. It copies each table once
Section titled “1. It copies each table once”When a pipeline first runs, the rows already in each table you picked are copied into your destination, in a schema the pipeline writes, unless you chose not to copy them (a pipeline that reads its source by queries always copies them). A copy that is interrupted carries on from where it had got, in most cases: A pipeline’s tables says when it cannot, and what each table’s state means while it is copied.
2. Then it brings over what changes
Section titled “2. Then it brings over what changes”How depends on how the pipeline reads its source, which you choose when you create it:
- From the change log (logical replication): every insert, update and delete arrives as your database writes it. This needs your database set up for it once: Set up a PostgreSQL source.
- By queries: each sync reads the rows that changed since the last one. It needs no change log, and it does not see rows deleted in your database: Reading PostgreSQL by queries.
3. Each table is kept the way you choose
Section titled “3. Each table is kept the way you choose”A table’s sync mode decides what a deleted row becomes in your destination, and whether every version of a row is kept: A table’s sync mode. You can leave a column out, hash it, or filter rows, before they reach your destination: Blocking, hashing and filtering.
4. When something it cannot decide happens, it asks
Section titled “4. When something it cannot decide happens, it asks”A change to a table’s shape at your source, a value your destination cannot take, or a row your destination does not keep: each has a setting for what the pipeline does, and when it stops a table it says why and what you can answer. When a source table’s shape changes, Values your destination cannot take, Rows your destination discards.
5. Everything it does is recorded
Section titled “5. Everything it does is recorded”Each run and each event is in the pipeline’s history, alerts tell you when it needs you, and the changes made to your account are in the audit log. History, runs and the overview, Notifications, The audit log.
6. You pay for what it used
Section titled “6. You pay for what it used”A pipeline is charged for the compute and memory its work used, from your wallet, and a run that ended in failure is not charged. Usage and billing.