Skip to content

Your first pipeline

A pipeline needs three things: a source to read, a destination to write into, and the tables to copy. This page goes through them in the order the app asks for them, and links to the page that has each one’s details.

You join an account by invitation: an owner invites your address, and you sign in with that address. Enter it on the sign-in page. If your organisation signs in with single sign-on, you are taken there; otherwise we email you a sign-in link, which works once. There is no password. If the address is not a member of any account, the link says so when you open it: sign in with the address the invitation was sent to. Members and invitations says how an owner invites someone.

Prepare your database and add it as a source: Set up a PostgreSQL source, step by step goes from turning on logical replication to a passed connection test, with what differs on each host. If your database cannot turn on logical replication, a pipeline can read it by queries instead, where your account offers it: Reading PostgreSQL by queries.

An owner adds destinations: under Destinations, press Add a destination and choose PostgreSQL or Snowflake. A Snowflake destination or A PostgreSQL destination says what to prepare and what is written there. An owner can also add one from the pipeline’s Destination step without leaving it; a member is told that only an owner can add one.

Under Pipelines, press New pipeline. Its steps, in order:

  1. Source: how to connect it (Use a saved source, Paste a connection URL or Enter the details), the Pipeline name, and, where your account offers both, How we read your database: from its change log, or by queries.
  2. Connection check: your database is tested, as the source form’s test does; a step that fails says why and what to change. A saved source whose own test passed skips it.
  3. Destination: one of your account’s destinations, or (for an owner) a new one.
  4. Pipeline check: your database and the destination are checked together, with the pipeline’s own names.
  5. Choose data: the tables to sync and, for each, Choose columns to leave a column out, hash it or filter its rows (Blocking, hashing and filtering).
  6. Sync settings: Rows that are already there: Copy my existing rows first, or Sync only changes from now on, and nothing is chosen for you (a pipeline that reads by queries always copies them first); its schedule, where one applies; and, under More settings, what the pipeline does when a source table’s shape changes (When a source table’s shape changes) and with a value your destination cannot take (Values your destination cannot take).
  7. Review and create: what the pipeline will copy, from where to where.
  8. Prepare your database: the statements to run on your source before it starts. We never run them ourselves; Send to your admin gives them to the person who can.
  9. Start: create the pipeline’s change bookmark (replication slot) right before you start, then Start. Start opens once a check finds the published tables (publication) and the change bookmark.

Steps 8 and 9 are step 5 of the setup guide, with the statements as the app shows them. A pipeline that reads by queries has neither: it needs no publication and no change bookmark, and once it is created the app takes you to its page.

The pipeline’s page has its Overview, its tables on Schema, what happened on History, what it used on Usage, and every setting on Settings. A pipeline’s tables says what each table’s state means, and History, runs and the overview what is recorded. When something stops a table or the pipeline, its page says why: Messages, and where they are explained lists every message with where it is explained.