Skip to content

Pipeline settings

A pipeline is created with POST /v1/pipelines, whose body has the 19 fields below. Each says whether it can be changed once the pipeline exists, and with which request.

  • Type: ChangePolicyRequest, optional.
  • Changed after creation: PUT /v1/pipelines/{id}/change-policy.
  • Explained in: this guide.

What the pipeline does when a source table’s shape changes: a preset, rules, a confirmation. Left out means the automatic preset, unconfirmed: a table whose key changes, or that is replaced, waits for you.

  • Type: boolean, optional.
  • Changed after creation: no request in the API changes it.

Whether you ran POST /v1/connections/test for this pipeline and it passed. A report, not a proof; stored and shown. Left out means false.

  • Type: string, optional.
  • Changed after creation: no request in the API changes it.
  • Explained in: this guide.

A credential reference holding a Postgres destination’s connection string (the older form). Sent with source.

  • Type: string, optional.
  • Changed after creation: no request in the API changes it.

A configured destination, from POST /v1/destinations. Sent with source_id.

  • Type: string, optional.
  • Changed after creation: no request in the API changes it.
  • Explained in: this guide.

A pipeline writes one schema in its destination, named once when the pipeline is created and never renamed. destination_schema_naming chooses how. fivetran, the default for a Snowflake destination, names it <pipeline id>_<source schema>: lower-cased, every character outside a-z, 0-9 and _ replaced by _, a leading digit given a _ in front, and a name over 63 bytes cut to 54 and ended with _ and 8 hex digits of a hash. source uses the source schema’s name exactly. A name beginning pg_ or _reyansh_, or information_schema, and a source name that is too long for the destination or begins or ends with whitespace, is refused with 422 invalid_destination_schema, naming the rule and the name. The choice is not offered for a Postgres destination yet: such a pipeline writes public, and a request that sends the field is refused with 422 unsupported_destination_schema_naming. GET /v1/destinations/{id}/schema-preview answers the name a pipeline would get before you create it.

  • Type: string, optional.
  • Changed after creation: no request in the API changes it.

Whether rows that predate the pipeline are copied on its first run. Leave it out if you have not decided: a first run that needs the copy then stops and asks, and never guesses. A query_based pipeline copies every table on its first run, so it records requested when this is left out, and refuses declined with 422 historical_load_needs_logical_replication.

  • Type: string, required.
  • Changed after creation: no request in the API changes it.

The pipeline’s name, unique in your account: letters, numbers, dashes and underscores, from 1 to 63 characters, or it is refused with 422 invalid_pipeline_name; new, in any case, is refused with 422 id_reserved (it is kept for the app’s page that adds a pipeline, /pipelines/new). It is not changed later, and it is not reused after the pipeline is deleted. A pipeline created before this rule keeps its name, and every other endpoint still finds it by it.

  • Type: string, optional.
  • Changed after creation: PUT /v1/pipelines/{id}/discarded-rows.
  • Explained in: this guide.

“continue” (the default for a new pipeline) reports each row your destination accepts and does not keep (a trigger or rule that discards it) on the activity feed and in an alert, and keeps replicating. “stop” reports it and stops the table that holds it, without saving that table’s progress past the row, so once the destination is fixed and the table resumed, the row is written. A table kept in history or append is answered the same way: a version or log row its destination discards is reported, and “stop” stops that table; a trigger that skips the pipeline’s own update or delete of an earlier version is not detected. The pipeline’s other tables keep syncing, unless the pipeline is set to stop every table when one table stops (when_a_table_stops). Until the pipeline has saved its place once (a new pipeline that copies nothing first, at its first batch), there is no place to keep the table at, so the pipeline stops instead, until you start it again. Left out means continue.

  • Type: string, optional.
  • Changed after creation: PUT /v1/pipelines/{id}/discarded-rows.
  • Explained in: this guide.

What happens when your destination refuses a VALUE. Only a PostgreSQL destination reads this answer; what a Snowflake destination does instead is the last sentence. For a PostgreSQL destination: “continue” (the default): a text value longer than its varchar(n) or char(n) column is cut to the column’s length in characters, and a value that is not a valid uuid or JSON for its column is written as NULL where the column allows NULL. A json column is written as json, so it keeps the value as sent; a jsonb column refuses a \u0000 escape and an unpaired surrogate escape, which are written as NULL there. A cut or a NULL is made only after the destination has refused the row, never to a key column, and is reported as a values_fitted event; syncing carries on. A table kept in history or append is answered the same way: its batch is written in one transaction, and sent once more with each such value changed. A NUL character (U+0000) in a value for any column but bytea, which a MySQL, Stripe or platform history source can send and no PostgreSQL text holds, is removed before the row is sent, because PostgreSQL refuses it before it reads the row; the rest is written as sent (reason nul_character), never in a key column, and reported as a values_fitted event too, and the record keeps its original as hex. Under either answer, a row the destination refuses for any other value (a value in a key column, a NULL where the column allows none, a number wider than its column keeps, a NUL character in a key column) or for a constraint or a trigger stops the table that holds it, which keeps its place, and the pipeline’s other tables keep syncing, unless the pipeline is set to stop every table when one table stops (when_a_table_stops). A refusal that passes by itself (a lock not granted in time, a timeout, a lost connection) restarts the pipeline on its own instead. A timestamp the source sends in a form other than ISO, or with a UTC offset other than zero for a timestamp column, is every table’s, so it restarts the pipeline on its own too. A value refused from the platform history export, our own source, is a fault on our side, and restarts the pipeline on its own. “stop”: nothing is cut, NULLed or has a NUL removed. A row the destination refuses for a value “continue” would cut or NULL, or holding a NUL “continue” would remove, stops the table that holds it, by the same rule, so the pipeline’s other tables keep syncing unless when_a_table_stops says to stop every table: the table waits for you, nothing of it past the row is saved, and the stop names the table, each such column and the reason (too_long or not_a_value), never the value. The row is not retried, so it may hold something else the destination refuses, which shows only afterwards. The stop is on the activity feed as table_parked, with the cause value_refused, and on the table with its remedy: change the column, or choose “continue”, then resume the table; where the destination database’s encoding refused a character, no column change helps, and neither does a re-sync: a PostgreSQL destination must be a UTF8 database, and a pipeline does not start on one in another encoding. A table stopped while it is copied stops alone, whatever when_a_table_stops says, and lets its place go, so resuming it copies it again. Until the pipeline has saved its place once (a new pipeline that copies nothing first, at its first batch), there is no place to keep the table at, so the pipeline stops instead, as schema_stop_raised, until you start it again. A pipeline whose source is MySQL or Stripe keeps one table and cannot stop it alone, so where its table would stop for a refused value, the pipeline stops instead, as schema_stop_raised, until you start it again; for a NUL the way out is “continue”, since that source reads the same change again whatever you correct at it. (The platform’s own history keeps one table too, but a value refused from it is ours, as said above.) A Snowflake destination does not read this answer: under either one, a value Snowflake cannot hold is written as NULL, the row’s _REYANSH_META column keeps a note of the original, told as a values_fitted event (not yet in a table kept in history or append, where only the note is kept). Never in a key column: a row whose key Snowflake cannot hold is not written and stops the table that holds it until you act, as does a number with more digits than its NUMBER column holds, which is never rounded; the pipeline’s other tables keep syncing, unless when_a_table_stops says to stop every table. Left out means continue.

  • Type: string, optional.
  • Changed after creation: POST /v1/pipelines/{id}/resync.
  • Explained in: this guide.

The publication name a connection test returned in setup. See slot_name.

  • Type: boolean, optional.
  • Changed after creation: PUT /v1/pipelines/{id}/discarded-rows.
  • Explained in: this guide.

Whether the pipeline keeps a record in YOUR destination: the keys of rows your destination discards, and the original of every value cut or NULLed. true (the default): for a PostgreSQL destination they are written to a table named _reyansh_rejected_v1 in the schema the pipeline writes, created the first time there is something to record. If the role may neither create that table nor insert into it, syncing carries on, and the rows_discarded or values_fitted event says the record was not kept and how to allow it. false: nothing is written or created, and what is not recorded while it is off cannot be recovered later, because we never store it. Left out means true.

  • Type: ScheduleRequest, optional.
  • Changed after creation: PUT /v1/pipelines/{id}/schedule.

How often the pipeline syncs. Left out means every six hours.

  • Type: string, optional.
  • Changed after creation: POST /v1/pipelines/{id}/resync.
  • Explained in: this guide.

The replication slot name a connection test returned in setup for this pipeline. Sent with publication_name, or not at all: left out, names are drawn here. Only a pair a test returned is accepted.

  • Type: string, optional.
  • Changed after creation: no request in the API changes it.

A credential reference holding the source’s connection string (the older form). Sent with destination.

  • Type: string, optional.
  • Changed after creation: no request in the API changes it.
  • Explained in: this guide.

A configured source, from POST /v1/sources. Sent with destination_id.

  • Type: array of string, optional.
  • Changed after creation: no request in the API changes it.

What the pipeline syncs, by name, without a schema; each is read as a table in public. Send this or tables, not both (tables_and_streams). A schema-qualified name is refused with invalid_stream_name. A PostgreSQL source takes several; a MySQL or Stripe source syncs one, and more is refused with one_stream_per_pipeline.

  • Type: string, optional.
  • Changed after creation: PUT /v1/pipelines/{id}/sync-mode.
  • Explained in: this guide.

The sync mode the pipeline’s tables are kept in, recorded on each table now and on each table added later: soft_delete, live, history or append (the last two only for an account they are available to: 422 not_yet_available otherwise). Absent: the destination’s default (Snowflake soft_delete, PostgreSQL live). 422 mode_not_offered for another.

  • Type: array of TableAsked, optional.
  • Changed after creation: POST /v1/pipelines/{id}/resync, POST /v1/pipelines/{id}/tables.
  • Explained in: this guide.

For a PostgreSQL source: the tables to sync, each by its schema and name, in the order you want them copied when nothing else decides it. Send this or streams. At most 1,000 (too_many_tables); each name 1 to 63 bytes with no NUL (invalid_table_name); a table named twice is duplicate_table. Two tables the destination would write as one table (Snowflake folds names to upper case) are refused with destination_table_collision, and two source schemas it would write as one schema with destination_schema_collision, each naming both. Each table lands in the schema destination_schema_naming names for its own source schema; a Postgres destination writes it in the schema of the same name. Another source refuses this field (tables_need_postgres). The answer’s setup carries the statements that publish exactly these tables.

  • Type: string, optional.
  • Changed after creation: POST /v1/pipelines/{id}/resync.
  • Explained in: this guide.

How the pipeline reads its source. logical_replication (left out means this) reads the source’s log through a replication slot and a publication. query_based reads each table by queries, one run at a time on the schedule, where the source offers it (422 update_method_not_offered otherwise); it takes no slot_name or publication_name (422 source_names_need_logical_replication) and no history or append default mode (422 mode_needs_logical_replication). Offered to every account unless the deployment’s operator switches it off (query_capture_enabled on GET /v1/me); while off, query_based is 422 not_yet_available and nothing is created. Changed later only by a full re-sync.