When a source table's shape changes
A PostgreSQL pipeline follows changes to the shape of the tables it syncs: a column added, dropped or renamed, a type changed, a table truncated, a table added to the publication or gone from it. What it does with each is set by the pipeline’s change policy: one answer for each kind of change. Whatever the answer, the pipeline tells you each change on its history, and what it did.
The policy covers pipelines with a PostgreSQL source. A MySQL source is not covered (see Limits).
How the pipeline notices a change
Section titled “How the pipeline notices a change”- In the change log. PostgreSQL sends a table’s new shape before the first row change made to it after the shape changed. The pipeline compares that shape with the one it has delivered, at that exact place in the log, so the rows before the change are written in the old shape and the rows after it in the new one.
- When the pipeline starts. It reads each table’s shape from your source’s catalogue before it reads the log, so a change made while it was stopped is found then, before any row is written.
- While it runs. It reads each table’s shape again on its own. That finds what the change log alone cannot tell apart, such as a column dropped, renamed, added again or no longer published, within about a minute.
What counts as a change
Section titled “What counts as a change”| At your source | The rule that answers it |
|---|---|
| A column added | A column is added (column_added) |
| A column dropped | A column is removed (column_removed) |
| A column renamed | Both of the above: its old name is removed and its new name is added. By default the new name is added and its existing values copied, and the old one is kept with the values it had. |
| A column dropped and added again under the same name | A column is added. Adding it again reset its values at the source, so the destination’s values in it are replaced with the source’s. |
| A column the publication’s column list sends again, or a column renamed away and back | A column is added. Where the pipeline found the column still in the table when it stopped being sent (no longer published, or renamed), it is told as the same column, whose values the source kept; otherwise it is told only that the source sends it again. Either way the destination’s values in it are replaced with the source’s. |
| A column’s type changed in a way that may change the values its rows hold | The table stops and is copied again: no rule answers it (see A column’s type changes in a way that may change its values) |
| A column’s type changed in a way that keeps every value | One of three, by what the destination column can hold: A column’s type widens (type_widened) when the destination column cannot hold every value of the new type but a wider type of it can; A column’s type narrows (type_narrowed) when it already holds every value of the new type (with one exception on Snowflake, below); A column’s type changes otherwise (type_changed) when no wider type of it can. A change of length, precision or scale counts: varchar(50) to varchar(100), numeric(10,2) to numeric(12,2). |
| A column the publication’s column list stops sending | A column is removed (column_removed) |
| A column added to a table whose publication has a column list that does not name it | Told on the history, with how to add it to the list, and not synced: take the table out of the publication and add it back in one transaction, naming it ONLY both times, with every column the list sends now and the new one, keeping its row filter. Once the publication sends it, it is a column added. |
TRUNCATE |
A table is truncated (table_truncated), when the publication sends truncates |
| A table the publication sends that the pipeline does not sync | A new table appears in the publication (table_found) |
| A table dropped, renamed, or taken out of the publication | A table is dropped or renamed (table_gone) |
| A table’s primary key or replica identity changed | A table’s key changes (key_changed) |
| A table dropped and created again under the same name | A table is dropped and created again (table_replaced) |
| The publication stops sending a table’s updates or deletes, or sends only the rows a row filter keeps | The publication sends less of a table (publication_narrowed) |
One more rule is about the other tables: When one table stops (when_a_table_stops) says what
the pipeline’s other tables do when a table stops, for any reason but a few found as the pipeline
starts, which always stop that table alone (see When one table stops).
Where to set it
Section titled “Where to set it”-
A new pipeline: step 6, Sync settings: open More settings: When the source changes, and the rest. Choose a preset, or open Advanced: one answer for each kind of change. Beside each answer is a line saying what it does.
-
An existing pipeline: Settings › Schema changes on the pipeline’s page, then Save. A running pipeline restarts on its own to use the new answers, within a minute. Saving without a change restarts nothing.
-
One table: the table’s drawer on the Schema tab (press the table’s name) offers Ask me about every change to this table. It sets Ask me for every rule a table can set for itself that offers it; the table keeps the pipeline’s answer for the others. A table with rules of its own offers Use the pipeline’s policy instead, which clears them.
-
API:
GET /v1/change-policy: every rule, its answers with what each does, its default, whether a table may set it for itself, and the two presets.GETandPUT /v1/pipelines/{id}/change-policy: the pipeline’s answers (preset,rules,confirm).PUTandDELETE /v1/pipelines/{id}/tables/{table_id}/change-policy: one table’s own answers. A rule the table may not set is refused with422 rule_not_per_table.POST /v1/pipelinestakeschange_policy; left out, the pipeline has the automatic preset.POST /v1/pipelines/{id}/tables/{table_id}/resumetakesanswers(see Answering a table that waits).
Each write takes an
Idempotency-Key. An answer a rule does not take is refused with422 answer_not_offered. See the API reference.
The presets
Section titled “The presets”- Automatic (recommended): the pipeline follows the source’s changes on its own and tells you each one. Every rule has its default, below. A new pipeline has this preset unless you choose otherwise.
- Ask me about every change: a table stops when the pipeline meets a change to it, until you answer. A change found by reading your schema, such as a dropped, renamed or re-added column, is met within about a minute, and rows that arrive in that minute are written as they are. This preset sets Ask me for every rule that offers it, Tell me for a new table, Stop the table for a publication that sends less, and Keep it for a table dropped or renamed.
Neither preset changes When one table stops. A pipeline whose answers match neither preset reads Custom. A table’s own answers are shown in its drawer (press its name on the pipeline’s Schema tab), and do not change what the pipeline reads as.
The rules
Section titled “The rules”| Kind of change | Answers | Default | Ask me about every change | A table may set it |
|---|---|---|---|---|
A column is added (column_added) |
Add it (add), Leave it out (leave_out), Ask me (ask) |
Add it | Ask me | yes |
A column is removed (column_removed) |
Keep it (keep), Drop it (drop), Ask me (ask) |
Keep it | Ask me | yes |
A column’s type widens (type_widened) |
Widen the destination column (widen), Ask me (ask) |
Widen the destination column | Ask me | yes |
A column’s type narrows (type_narrowed) |
Keep it (keep), Ask me (ask) |
Keep it | Ask me | yes |
A column’s type changes otherwise (type_changed) |
Convert the destination column (promote), Ask me (ask) |
Convert the destination column | Ask me | yes |
A table’s key changes (key_changed) |
Copy the table again (recopy_table), Ask me (ask) |
Copy the table again, once you confirm it | Ask me | yes |
A table is dropped and created again (table_replaced) |
Copy the table again (recopy_table), Ask me (ask) |
Copy the table again, once you confirm it | Ask me | yes |
A table is truncated (table_truncated; a table with no primary key is emptied whatever it says, below) |
Mark its rows deleted (soft_delete), Delete its rows (delete), Keep its rows (ignore), Ask me (ask) |
Mark its rows deleted | Ask me | yes |
The publication sends less of a table (publication_narrowed) |
Sync what it sends (continue), Stop the table (stop) |
Sync what it sends | Stop the table | yes |
A new table appears in the publication (table_found) |
Add it (add), Tell me (notify) |
Add it | Tell me | no: the whole pipeline’s |
A table is dropped or renamed (table_gone) |
Keep it (keep), Stop every table (stop_every_table) |
Keep it | Keep it | no: the whole pipeline’s |
When one table stops (when_a_table_stops) |
Keep the others syncing (keep_others_syncing), Stop every table (stop_every_table) |
Keep the others syncing | unchanged | no: the whole pipeline’s |
- Leave it out is for a column the source adds: the destination table never gets it, and the pipeline takes it out of every row of that table, a copy’s rows included. A key column cannot be left out; for one, the table asks. Leaving out a column the table already syncs is not offered.
- The three rules a table may not set decide what is added to the pipeline, or what the other tables do, so they belong to the pipeline.
Ask me
Section titled “Ask me”When the answer is Ask me, the table stops when the pipeline meets the change, before it writes anything of that table past it. If the table has a place in the source’s change log, that place is kept while it waits, so nothing is lost: once you answer, it carries on from where it stopped. While it waits, that place keeps the source’s change log on your source’s disk, and if your source runs short of room for it, the place is let go. A table that has no place yet (for example one still waiting to be copied when the pipeline starts and finds the change) and a table whose place was let go are both copied from the start once you answer, each by its mode: in soft delete keeping every row your destination already has, a row your source no longer has marked deleted; kept live, replacing what your destination holds, a row your source no longer has removed, unless the table has a primary key and its destination is PostgreSQL, which keeps it, and a row a new row filter leaves out is, by the table’s mode and your answer to that filter’s change, kept, marked deleted or removed; that copy is billed like any other work.
A changed key and a replaced table are different: there, Ask me lets the table’s place go. The table stops at the change and waits until you resume it, and nothing carries on from where it stopped: Resume copies it again, by its mode: in soft delete keeping every row your destination already has, a row your source no longer has marked deleted; kept live, replacing what your destination holds, a row your source no longer has removed, unless the table has a primary key and its destination is PostgreSQL, which keeps it, and a row a new row filter leaves out is, by the table’s mode and your answer to that filter’s change, kept, marked deleted or removed; that copy is billed like any other work, and the table that has the name then is copied as your source keys it. This is what Copy the table again does too while it is not confirmed (see A table’s key changes, or the table is dropped and created again).
The pipeline’s other tables carry on, unless When one table stops is Stop every table: then every other table that is syncing stops with it, whether the change is met while the pipeline runs or found when it starts (see When one table stops).
Copying a table again on its own needs your confirmation
Section titled “Copying a table again on its own needs your confirmation”Copy the table again is the default for a changed key and for a replaced table, but it acts only once you confirm it. While a rule’s answer is Copy the table again, the editor shows:
I confirm: when a table’s key changes or it is replaced at the source, the pipeline copies it again on its own, by its mode: in soft delete keeping every row your destination already has, a row your source no longer has marked deleted; kept live, replacing what your destination holds, a row your source no longer has removed, unless the table has a primary key and its destination is PostgreSQL, which keeps it, and a row a new row filter leaves out is, by the table’s mode and your answer to that filter’s change, kept, marked deleted or removed, and that copy is billed like any other work.
Until it is ticked and saved, the editor says “Not confirmed: a table whose key changes, or that is replaced, waits for you.” A pipeline created before the policy existed has no confirmation.
A confirmation records who gave it and when. It belongs to that rule, at that level (the pipeline,
or one table), and any later write of that rule that does not confirm again removes it:
choosing a preset, or setting the rule, even to Copy the table again again. Through the API, a
write confirms with confirm: true.
What each answer does, at each destination
Section titled “What each answer does, at each destination”A Snowflake destination writes the pipeline’s own schema, so the pipeline changes its tables itself. A PostgreSQL destination table is yours, so the pipeline changes it only where the change is small and safe, and otherwise shows you the statement to run. Its role may change your table only when it owns the table, or is a member of the owning role that inherits its privileges. Without that, every change below that alters the table waits with the statement, which names the role to run it as.
A column is added
Section titled “A column is added”| Answer | Snowflake | PostgreSQL |
|---|---|---|
| Add it | The column is added, then its existing values are copied from the source (below). | The same, when the role may alter your table. The column gets the source’s type: a built-in type by its name, an enum as text, a domain as its base type, an extension’s type only if your destination database has that type. Otherwise the table waits with the ALTER TABLE ... ADD COLUMN statement: run it, then answer Add it and copy its values. |
| Leave it out | Not added; rows carry nothing of it from then on. | The same. |
| Ask me | The table waits. Answers: Add it and copy its values, Leave it out. | The same. |
Copying an added column’s existing values (a fill). Rows that were in the source table before the column was added may hold values in it that the change log never carried: a default, an identity, a generated value, or a renamed column’s values. So the pipeline reads the column from your source and writes each value into the destination row with the same key. It only updates rows that are there: it never adds a row, and never brings back a row marked deleted. While it runs, the pipeline’s other tables wait, then catch up. The fill is metered like a table’s copy.
A fill first waits for the transactions writing the table, as a copy does: a value a transaction wrote before the pipeline sent the column could otherwise be missed. If one is still open after one second, that table alone pauses at its place, its drawer names the sessions (by process id), and the other tables stream. The pipeline fills the column on its own once the transactions it first saw have ended, also after a restart: a transaction that began later is never waited for, so a table that is never quiet is still filled. Nothing needs doing; ending a transaction left open by mistake is all it takes.
The pipeline skips the fill only when your source’s catalogue proves the existing rows hold nothing
in the column: for example a plain ADD COLUMN with no default, on an ordinary table, met in the
change log after the pipeline last read the table’s shape. Otherwise it fills.
A column dropped and added again under the same name is filled the same way, over every row, so the
values your destination held in it become the source’s, NULL included.
A column is removed
Section titled “A column is removed”| Answer | Snowflake | PostgreSQL |
|---|---|---|
| Keep it | The destination keeps the column and the values it has. New rows leave it empty, and an update leaves it as it was. | The same, while the column allows NULL in your table. If it is NOT NULL there, every new row would be refused, so the table waits with the ALTER TABLE ... ALTER COLUMN ... DROP NOT NULL statement: run it, then answer Keep it and its values. |
| Drop it | The column is dropped from the destination table, only while the source has no column of that name. | Never dropped by the pipeline: the table waits with the ALTER TABLE ... DROP COLUMN statement. Run it, then answer I dropped it. |
| Ask me | The table waits. Answers: Keep it and its values, Drop it from the destination. | The table waits. Answers: Keep it and its values, I dropped it. |
A column’s type changes in a way that may change its values
Section titled “A column’s type changes in a way that may change its values”To change a column’s type, PostgreSQL rewrites every row of the table, and it sends none of the
rewritten rows through its change log. Where the new type stores a value differently, your
destination still holds the value as it was: timestamp(6) made timestamp(3) rounds
10:00:00.123456 to 10:00:00.123, a numeric given a scale rounds 1.5 to 2 and writes
12345.0000 as 12345, timestamp made timestamptz adds the offset. Nothing the source sends
later would correct those rows. So, for a table with a primary key, whatever the rules say, the
table stops syncing: it parks with the cause type_change_rewrote, nothing of it after the change
is written, and the pipeline’s other tables carry on. Its drawer says what changed, what a stale value
does, and what to do:
| The column | What a stale value does |
|---|---|
| Part of the table’s key | Your destination no longer finds the row by its key: a later change to it would be written as a second row, and its delete would miss it. |
| Any other column | The row holds the old value. |
A table with no primary key is never stopped as type_change_rewrote (since 2026-10-29 (IST)).
Each of its rows is identified by a hash of all its values, so such a change gives every row a new
identity, and the table is copied again under the new ones by its rule for a change of its key, as a
dropped column does: at once under a confirmed Copy the table again, otherwise stopped (cause
replica_identity_key_changed) until you resume it, which copies it; a change found as a Snowflake
pipeline starts stops it either way. The column’s type is changed first, as the rules below say.
That copy replaces what your destination holds (kept live) or marks deleted every row it does not
write (soft delete), so it ends exact: there is nothing to choose.
Its copy still waits for a column that cannot hold the new type (since 2026-10-30 (IST)). A copy
into a PostgreSQL table writes into its columns as they are declared, whatever the table’s key, so
where your column cannot hold every value of the new type (a numeric(10,2) given numeric(12,4)
values would keep 1.23 of 1.2345), a copy of a table with no primary key (that re-key, a
Re-sync, or a resume) copies nothing and the table stops with the cause
copy_needs_column_type. Its drawer ends with the statement that changes the column: run it, then
Resume it. The pipeline checks that the column has the new type, or a type that one changes to in place
(the same kind, wider), and only then copies
the table again, and its drawer says what that copy does, as every copy of this table does: it
replaces what your destination holds, so a row your source no longer has, your own rows among them,
is removed (kept live), or it marks such a row deleted (soft delete). Before 2026-10-30 such a table
was stopped as type_change_rewrote and told that its copy keeps every other row your destination
holds, which is true of a table with a primary key only.
What brings a table with a primary key back is a copy from the source as it is now, and what that copy needs is the destination’s:
-
Snowflake. The table’s place in the change log is let go. Resume it: the copy changes the destination column’s type itself, writes every row again, and the rows your destination holds that the source does not have, among them any written with a rewritten key’s old values, are marked deleted (deleted under hard delete).
-
PostgreSQL. The table keeps its place in the change log while it waits for you (where it had one; that place keeps the log on your source’s disk), but it never carries on from there: it comes back by a copy. A copy writes into your table’s columns as they are declared and never changes a column’s type, and a column whose type cannot hold every value of the new one would have values rounded or cut (a
numeric(10,2)column given1.2345keeps1.23; adatecolumn given10:30keeps midnight). So where your column cannot, the table’s drawer ends with theALTER TABLE ... ALTER COLUMN ... TYPE ...statement that changes it to the source’s new type: run it first. How it converts the values already in your table does not need to match how your source converted its own, which the change log does not show: the copy writes every value again. Then bring it back by a copy: the pipeline checks that the column has the new type, or a type that one changes to in place, which holds every value of it (acharacter varyingof greater length or none, ortext; anumericof greater precision and the same scale, or none; atimestampof finer precision or none; anintervalof the same fields and finer precision, or none), and only then copies the table again: anumeric(14,4)column is taken for anumeric(12,4)one. Before 2026-10-30 the table’s line said that only the new type, exactly, was taken, which was never so. Which copy depends on the column, and the table’s drawer says which:- A column of the table’s key (cause
type_change_rewrote_key): the table cannot be resumed; choose one of two copies, because only you know which rows your table should keep. The table’s drawer offers both, and no Resume. Re-sync (copy it again in its drawer;POST .../recopy) keeps every other row your table holds, the rows written with the key’s old values among them, beside the new ones. Re-sync and delete them (copy it again, deleting rows not in the source;delete_rows_not_in_source) deletes every row your table holds that the copy does not write, the rows written with the key’s old values and your own rows among them, but where the table keeps rows outside its row filter, or you changed that filter since it last synced, it deletes none of the rows outside it, and where your table’s values cannot be judged against that filter (it reads a column this pipeline hashes, or one your table declares differently from your source), it deletes nothing, and the table’s drawer says so. It is offered here before any copy has counted those rows, so its confirmation says what it deletes, never a number. A resume is refused with409 copy_choice_requiredwhatever the table’s row filter says, and changes nothing. - Any other column: Resume it; the copy writes every row again and keeps every other row
your table holds (the resume is recorded as a plain
recopy). The one thing it deletes is what you asked a row filter to delete: if you changed the table’s filter answering that the rows outside it are deleted, this copy applies that answer. - A table kept with deleted rows marked (
soft_delete) is resumed either way, and its copy marks the rows this pipeline wrote that the source does not have as deleted instead.
Until the column has the new type or such a type, a resume or a Re-sync copies nothing: the table parks again, saying so. Where your column already holds every value of the new type (a
timestamp(6)column whose source column becametimestamp(3)), there is no statement: resume it. Such a column keeps its own way of writing a value the copy finds equal: anumerickey holding12345.0000is the source’s12345, so that row is updated where it is and still reads12345.0000. - A column of the table’s key (cause
A copy is billed like any other work. A confirmed Copy the table again for key changes does not copy such a table on its own: that confirmation is for key changes.
What keeps every value, and so is answered by the rules below as before (an existing pipeline
whose source makes one of these changes sees nothing new). The same list decides whether a table
with no primary key is copied again under new identities: a change outside it is, and so is one in
it that makes the column another type, because such a table’s destination column is never changed
to another type in place. Only these keep it as it is: a varchar(n) made longer, varchar or
text; a varchar made text; a smallint or integer made a wider integer; a numeric(p,s)
given at least as many digits at the same scale; a timestamp or timestamptz given the same or a
finer precision (no precision is 6).
| From | To |
|---|---|
varchar(n) |
varchar(m) with m at least n, varchar, or text |
varchar |
text; and text to varchar |
numeric(p,s) |
numeric(p2,s) with p2 at least p (the same scale), or numeric |
smallint |
integer or bigint; and integer to bigint |
smallint, integer, bigint |
numeric, or numeric(p,0) |
smallint, integer, bigint, numeric |
text or varchar |
timestamp(p), timestamptz(p), time(p), timetz(p) |
the same type with a precision at least p, or none |
Anything else may change a value or the text the source sends for it, including a column whose type is a domain or an array, and the table stops. Each row above, and a list of changes that look like it and are not, is checked against PostgreSQL 17 and 18 by the pipeline’s own tests.
A column’s type widens
Section titled “A column’s type widens”| Answer | Snowflake | PostgreSQL |
|---|---|---|
| Widen the destination column | Only a NUMBER widens: it becomes NUMBER(38, s) at the same scale s, and nothing it holds changes. VARCHAR and BINARY columns are created at their largest, so they never need to. |
Changed in place only where PostgreSQL does it without rewriting your table or rebuilding an index: a varchar(n) made longer, unlimited or text; a numeric(p,s) given more digits at the same scale, or made unconstrained; a timestamp, timestamptz or interval given a finer precision, or none. Anything else waits with the ALTER TABLE ... ALTER COLUMN ... TYPE statement: run it, then answer I changed it. |
| Ask me | The table waits. Answer: Widen the destination column. | The table waits. Answer: I changed it. |
A column’s type narrows
Section titled “A column’s type narrows”| Answer | Both destinations |
|---|---|
| Keep it | Nothing changes at the destination: the column already holds every value of the new type, with one exception. On Snowflake a VARCHAR column holds any value, and a TIMESTAMP_NTZ column holds a date from year 1 to the year 294276: a date before year 1 or after the year 294276 is written as NULL, with its value kept in _REYANSH_META (see Values Snowflake cannot hold). The change’s line in the pipeline’s history names those dates when it applies, and so does a waiting table’s drawer under Ask me. |
| Ask me | The table waits. Answer: Leave the column as it is. |
A column’s type changes otherwise
Section titled “A column’s type changes otherwise”| Answer | Snowflake | PostgreSQL |
|---|---|---|
| Convert the destination column | Converted to a type that holds both the old values and the new (below), every value checked; it becomes the table’s last column. | Never converted by the pipeline, because the change would rewrite your table: the table waits with the ALTER TABLE ... ALTER COLUMN ... TYPE ... USING statement. Run it, then answer I changed it. |
| Ask me | The table waits. Answer: Convert the destination column. | The table waits. Answer: I changed it. |
On Snowflake, what a column is converted to:
| The destination column | Becomes |
|---|---|
NUMBER, when the new type needs more decimal places |
NUMBER(38, the new scale), when 38 digits hold both; otherwise VARCHAR |
NUMBER, otherwise |
VARCHAR |
DATE, when the source now sends a timestamp (without time zone) |
TIMESTAMP_NTZ(9) |
| any other type | VARCHAR |
The pipeline adds the new column, copies every value into it, and checks, row by row, that each
value is unchanged and converts back exactly. Only then does it swap the names and drop the old
column. Every conversion it uses was checked against Snowflake beforehand; a pair of types without
such a conversion is not converted, and the table asks you instead. If any value would change,
nothing is dropped. While it converts, the table briefly holds a working column whose name begins
_REYANSH_NEW_ or _REYANSH_OLD_: do not write to it or drop it.
If a conversion stops part way (the pipeline restarted, or Snowflake refused one of its steps and
the table stopped and told you), nothing is lost. The next time the table starts, after a restart
or after you resume it, the pipeline finishes the conversion from where it stopped, or puts the
original column back under its name and converts it again, before it writes anything to the table.
One case it does not finish yet: a stop after the converted column took the column’s name and
before the old one was dropped leaves a column whose name begins _REYANSH_OLD_ beside the
converted column. It holds the column’s values from before the conversion, every one of which was
checked against the converted column before the names were swapped, and the pipeline no longer
writes to it. You can leave it, unless the column was one of the table’s key columns: then the
_REYANSH_OLD_ column is NOT NULL, and the table stops at its next start and says so; drop that
column, then resume the table. Otherwise, if the source later changes the same column’s type in a
way that needs another conversion, the table stops and tells you; dropping that _REYANSH_OLD_
column, then resuming the table, lets the conversion run.
A table’s key changes, or the table is dropped and created again
Section titled “A table’s key changes, or the table is dropped and created again”| Answer | Snowflake | PostgreSQL |
|---|---|---|
| Copy the table again, confirmed | The table goes back to waiting and is copied again on its own: the rows your source no longer has are marked deleted in soft delete and removed in live, with or without a primary key. The old key’s columns stop being NOT NULL. |
The table is copied again on its own. Rows your source no longer has are marked deleted in soft delete; kept live, they are kept and counted if it has a primary key when it is copied (see Rows your source no longer has), or removed if it has none. Your table needs a unique index on the new key. |
| Copy the table again, not confirmed, or Ask me | The table stops and lets its place go. Resume copies it again: the rows your source no longer has are marked deleted in soft delete and removed in live, with or without a primary key. | The table stops and lets its place go. Resume copies it again: in soft delete a row your source no longer has is marked deleted; kept live, it is kept if the table has a primary key when it is copied, and removed if it has none. |
If a table’s key changes and the table is then dropped, renamed away, replaced or taken out of the publication before the pipeline has copied it again, the changes made to it before the key change still arrive: the pipeline writes them, then stops the table. One exception: if the pipeline itself stops abruptly (its machine fails) after it has read the key change and before it has saved the batch that holds it, the next run reads the table by its new key, cannot read the changes still in the log made under the old one, and stops the table at once; those changes, made between the pipeline’s last saved place and the key change, do not arrive, and the table’s drawer says so. The changes made after the key change cannot be read under a key the destination no longer agrees on, and do not arrive, which the table’s drawer says. Nothing of the transaction that changed the key is written (one larger than a batch may have had a first part written before the change was met). The pipeline offers a copy only of a table it finds at the source: a dropped table’s drawer says to resume it if a table of that name comes back, or to remove it; a table whose name now belongs to another table says that resuming copies that other table; and one the publication no longer lists gives the statement that adds it back first. Under the confirmed answer none of these is copied again on its own.
A table is truncated
Section titled “A table is truncated”A table with no primary key is emptied whatever the answer: every row its destination table held at the truncate
is removed when the table is kept live, and marked deleted (_REYANSH_DELETED_AT, _reyansh_deleted_at) when it
is kept in soft delete, and rows written after the truncate are live. It never waits at a truncate. The answers
below are a table with a primary key’s.
| Answer | Snowflake | PostgreSQL |
|---|---|---|
| Mark its rows deleted | Every row the destination table held at the truncate is marked deleted (_REYANSH_DELETED_AT is set). Rows written after the truncate are live. |
A table kept in soft delete is marked the same way (_reyansh_deleted_at is set), and rows written after the truncate are live. Any other table waits. Answers: Delete its rows, Keep its rows. Soft delete is not yet available on PostgreSQL, so for now every PostgreSQL table waits. |
| Delete its rows | Every row the destination table held at the truncate is deleted. | The same, with DELETE, never TRUNCATE. |
| Keep its rows | Nothing is deleted. | The same. |
| Ask me | The table waits at the truncate. Answers: Mark its rows deleted, Delete its rows, Keep its rows. | The table waits. Answers: Delete its rows, Keep its rows, and Mark its rows deleted for a table kept in soft delete. |
A table kept in history mode or append-only keeps every row it holds, at both destinations, and never waits for a soft delete it does not have. Under Mark its rows deleted or Delete its rows, a history table closes every current version, marked deleted at the truncate, and removes none; an append-only table adds one row of its own for the truncate. Under Keep its rows nothing changes, and under Ask me the table waits for your answer (Re-sync and sync modes has each mode’s rows).
No sentence about a truncate gives a row count.
The publication sends less of a table
Section titled “The publication sends less of a table”- Sync what it sends: the pipeline syncs what the publication sends, and the table carries a note saying what does not reach the destination, for example that delete changes made at the source do not, or that an update moving a row out of a row filter arrives as that row’s delete.
- Stop the table: the table stops. Resume copies it again.
Two things still stop the table, whatever the answer: a publication that does not send one of the
table’s key columns, and a column list on a table whose replica identity is FULL.
A new table appears in the publication
Section titled “A new table appears in the publication”- Add it: the table is added to the pipeline, and copied with its existing rows when the pipeline next starts, which a running pipeline does on its own within a minute. A pipeline created with “Sync only changes from now on” adds and copies it the same way: that answer was about the tables picked when the pipeline was created, and a table added to the publication later follows this rule, as does one of those tables renamed at the source, which is a new table to the pipeline. Set it to Tell me to decide each one yourself. If the add is refused (for example, another pipeline already writes that destination table), it is not added, and the history says why. A table you removed from the pipeline is never added back.
- Tell me: nothing is added. The table is listed under Tables found in the publication, with a button, Add {table}, until it is added.
A table is dropped or renamed
Section titled “A table is dropped or renamed”- Keep it: that table stops and its destination table keeps its rows. The other tables carry
on, unless When one table stops is Stop every table, which stops them for this as for any
other stop. A dropped table: if a table of that name comes back, Resume copies it; otherwise
Remove it. A renamed table: under its new name it is a new table to the pipeline, answered by
A new table appears in the publication. A table taken out of the publication: its drawer gives
the statement that adds it back; run it, then Resume it. PostgreSQL does not send changes
made to a table while it is out of the publication. When a table’s entry in the publication is
replaced between two looks (the pipeline looks about once a minute while it runs, and when it
starts) by one with the same column list and row filter, the table stops, keeping its place.
Two things leave exactly that: the table taken out of the publication and put back, or its column
list or row filter changed and changed back in two statements (
SET TABLEtwice). PostgreSQL’s catalogue does not say which, so the table’s drawer says both: if it was out, the changes made while it was out did not arrive. If you know it was never out, Resume it: it streams on from its place and nothing is copied. Otherwise a Re-sync brings its destination table back in line. While it waits, its place keeps the source’s change log on the source’s disk, as any table that keeps its place does. A list or filter you change while the pipeline runs is seen and recorded at its next look, so setting one and setting it back while it runs does not stop the table when a look falls between the two changes. Set and set back within one look (about a minute), the two read exactly as a table taken out and put back, and the table stops as above. A table put back with a different column list or row filter syncs on, as any change to its list does, and its history says that its entry in the publication was replaced: the pipeline cannot tell that change from a table taken out and put back with another list, so if yours was out in between, use Copy again on it. - Stop every table: that table stops, and so does every other table that is syncing, as described under When one table stops below.
When one table stops
Section titled “When one table stops”- Keep the others syncing: the other tables carry on.
- Stop every table: every other table that is syncing stops too, keeping its place, and resumes
on its own from there when the first is resumed, removed or copied again, whether you do it or
the pipeline does (it tries a table whose destination table another session holds again on its
own). This applies to a stop found when the pipeline starts as well as to one met while it runs,
except these, found as it starts, which stop that table alone while the others carry on, each
copied again when you resume it: a table whose copy at the start does not finish, because the
destination refused it for that table alone (for example, a value its column refuses while the
pipeline is set to stop on one, a column its table lacks, rows it discards while the pipeline is
set to stop on them, or a key that cannot be read back) or because copying it would hold more of the
source’s change log than the source keeps for this pipeline; and a table whose place someone
moved the pipeline’s replication slot past. A copy at the start is a new pipeline’s first copy,
or the copy of a table you added, re-synced (Re-sync), or resumed after its place was let
go.
- A table waiting to be copied, or being copied, carries on: it has no place to keep yet.
- Before the pipeline has saved its first place in the source’s change log, a table that stops with another has no place to keep either: it stops, does not resume with the first, and resuming it copies it again, by its mode: in soft delete keeping every row your destination already has, a row your source no longer has marked deleted; kept live, replacing what your destination holds, a row your source no longer has removed, unless the table has a primary key and its destination is PostgreSQL, which keeps it, and a row a new row filter leaves out is, by the table’s mode and your answer to that filter’s change, kept, marked deleted or removed; it is billed like any other work. Its drawer says so. And a stop found as the pipeline starts, before that first place is saved, stops no other table.
- A stopped table whose place is let go while it waits (your source ran short of room for its change log) does not resume with the first. It stays stopped until you resume it, which copies it again, by its mode: in soft delete keeping every row your destination already has, a row your source no longer has marked deleted; kept live, replacing what your destination holds, a row your source no longer has removed, unless the table has a primary key and its destination is PostgreSQL, which keeps it, and a row a new row filter leaves out is, by the table’s mode and your answer to that filter’s change, kept, marked deleted or removed; that copy is billed like any other work.
- Each stopped table’s drawer names the table it stopped with and says whether that table is still parked, and the first table’s drawer says how many tables stopped with it. Both are read from the tables when you look, never from this rule, so changing the rule later does not change what they say about a stop that already happened.
Answering a table that waits
Section titled “Answering a table that waits”A table that waits for you shows, in its drawer on the Schema tab, each change it waits on, such as “Column
loyalty_tier was added at the source.”, with one button per answer that change takes:
| Answer | Button | On a PostgreSQL destination |
|---|---|---|
add |
Add it and copy its values | |
leave_out |
Leave it out | |
keep, a column removed |
Keep it and its values | |
keep, a type narrowed |
Leave the column as it is | |
drop |
Drop it from the destination | I dropped it |
widen |
Widen the destination column | I changed it |
promote |
Convert the destination column | I changed it |
soft_delete |
Mark its rows deleted | |
delete |
Delete its rows | |
ignore |
Keep its rows |
Choose one for each change, then press Resume with these answers. It can be pressed only once every change the table waits on has an answer. The answers are applied when the pipeline next starts, which a running pipeline does on its own within a minute.
On a PostgreSQL destination, an answer that says you made the change is checked. Run the statement shown exactly as it is shown, then answer. The pipeline looks at your table, and if the change is not there yet, the table waits again and says so. The pipeline does not run the statement itself.
Through the API, POST /v1/pipelines/{id}/tables/{table_id}/resume takes
{"answers": {"<change_id>": "<answer>"}}, with each change’s change_id and the answers it takes
in pending_changes on GET /v1/pipelines/{id}/tables. A resume missing an answer is refused with
409 answer_required, naming each change still without one; a change_id that no longer waits
with 409 change_not_pending. A refusal’s message names each change by its column and what happened
at the source, and its details.changes carries each one’s change_id.
Re-sync on a waiting table copies it in the source’s shape as it is, without asking, and the
changes the table waited on are closed. The pipeline adds what the source added. A column whose type
the source changed is kept as it is where it already holds the source’s values, widened, or
converted to a type that holds them, whatever the type rules say. (A Snowflake TIMESTAMP_NTZ
column kept for a date writes a date before year 1 or after the year 294276 as NULL, as above.) On
Snowflake a converted column becomes the table’s last column. On PostgreSQL the pipeline widens a
column only where PostgreSQL does that in place, and otherwise writes into the column as it is: its
ALTER ... TYPE stays yours to run. It never drops a column, or deletes rows, that nobody answered
for.
A change can take no answer: one this destination cannot make at all, for example a type change of
a column your Snowflake table no longer has. Its table stops and shows no Resume; the other
tables carry on, unless When one table stops is Stop every table. Its drawer says to Copy
again (every row is read again, and the copy is billed like any other work) or Remove it.
Through the API such a change has answers: [], and a resume of its table is refused with 409 answer_required, saying the same.
What you see
Section titled “What you see”The history names each change and what the pipeline did. For example:
| Change | What the history says |
|---|---|
| A column added and filled | that it was added to the destination table and that its existing values are being copied while the other tables wait; then how many source rows were read, or that nothing needed copying |
| A column left out | that the pipeline leaves it out, so the destination table does not get it |
| A column added again | that the source reset its values, and the destination’s values are being replaced with the source’s |
| A column no longer sent | “The source no longer sends column … The destination keeps the column and the values it has.” |
| A column dropped at the destination | that it was dropped at the destination, as you answered |
| A type widened, narrowed or converted | the old and the new source types, and what changed at the destination; a converted column “is now the table’s last column” |
| A truncate | which answer was carried out, with no row count |
| A publication that sends less | what it sends now, and what does not reach the destination |
| A column a column list does not send | how to re-add the table with the column in its column list, keeping the table’s row filter and touching no table that inherits from it |
| A table found | whether it was added, not added (and why), or only told |
| A table dropped or renamed | that its destination table keeps its rows, and what to do |
The table’s drawer carries notes while they hold: “Copying the existing values of column …”, on the other tables “Waiting while the existing values of column … of … are copied.”, “Column … is left out of this table.”, and what a publication that sends less leaves out. A table that waits says what happened, what that means for the rest of the pipeline as it is when you look, and what brings it back. See A pipeline’s tables.
Alerts and notifications:
- Tables waiting for you (
table_parked, an error): one or more tables stopped, and nothing of them is written until they are resumed. Most wait for you. Three kinds come back without you, if they kept their place in the source’s change log: a table whose destination table another session held is tried again while the pipeline keeps running, a table whose column’s existing values the source made wait is read again while the pipeline keeps running, and a table that stopped with another resumes with that table. Sent two minutes after the newest one stopped, so one migration is one message naming every stopped table. table_gone(a warning): a table was dropped or renamed at the source; its destination table keeps its rows.- Schema changes handled (
schema_changed, a warning): the pipeline followed changes to its source tables’ shape. It stands for a day after the last one.
None is raised for a pipeline you stopped. See Notifications for where each is sent. The pipelines list shows, beside a pipeline’s health, how many of its tables wait for you.
Limits
Section titled “Limits”- A rewrite by an expression is not seen. A type change that may change values stops the table
(above), but the pipeline judges
the two types, never the statement:
ALTER TABLE ... ALTER COLUMN ... TYPE ... USING ...can rewrite values even between two types the table above keeps, or with the type unchanged, and so does a generated column’sALTER TABLE ... ALTER COLUMN ... SET EXPRESSION. PostgreSQL sends no rewrite through its change log, and a type that did not change is not seen at all. Your destination keeps the old values in every row the source has not changed since. After such a statement, use Re-sync on the table; where a key was rewritten, its drawer then counts the rows your table has that the source does not, and Re-sync and delete them removes them. - Only the type before and the type after are compared. Two
ALTER ... TYPEstatements with no row changed between them reach the pipeline as one change, from the first type to the last:numeric(10,2)madenumeric(10,0), which rounds every value, thennumeric(12,2), reads as a change that keeps every value, and the rounded values are never seen. After such a pair, use Re-sync as above. - An enum value renamed is not seen.
ALTER TYPE ... RENAME VALUErespells every row that holds the value, with no column’s type changed and nothing in the change log. Use Re-sync on each table with such a column. - A converted Snowflake column moves to the table’s last position. A query that reads columns
by position, such as
SELECT *into a fixed layout, sees the order change. - A column converted to
VARCHARcan hold one value spelled two ways, until 2026-10-26. A Snowflake column converted while the pipeline was catching up could hold Snowflake’s spelling (true,0A0B,+00:00) beside PostgreSQL’s (t,\x0a0b,+00). Those conversions now stop the table and copy it again, which writes every row one way; only a whole number or anumericmade text is still converted in place, and Snowflake spells those exactly as PostgreSQL does. A column converted before then keeps both spellings until it is re-synced (Re-sync). - A renamed table arrives as a new table. The pipeline does not follow it into its old destination table: under Add it the new name is copied in full into a destination table of its own name, and the old destination table keeps the rows it had. Remove the old name when you no longer need it.
- A fill never reads through a row-level security policy. If a policy on the source table
applies to the role the pipeline reads as, the fill would see only the rows the policy shows, so
the table stops before any value is copied, and its drawer names the role and the statement that
fixes it:
ALTER ROLE ... BYPASSRLS;, run as a superuser. A policy that shows the role every row is not enough, for the reason a table’s first copy gives. Then resume the table, and the column’s values are copied. - A fill waits for your source, briefly, and tries again later rather than fail. Before it reads,
the fill takes the lock a
SELECTtakes on the source table (ACCESS SHARE), so anALTER TABLEthat rewrites the table cannot leave it reading nothing; and it first waits, at most a second, for every transaction that may have changed the table before the pipeline sent the change (and, behind a synchronous standby, for a commit of the table the standby has not confirmed), so it never writes a row’s older value over a newer one the pipeline already has. Past that second the table pauses, keeping its place, and its drawer names the sessions and why; the pipeline reads the values on its own once those are over, also after a restart. If the table’s lock is not granted within 30 seconds instead, the table stops, keeping its place, and its drawer says so: while the pipeline keeps running it tries again on its own every five minutes, and once more each time it starts. The other tables carry on, unless the pipeline is set to stop every table. - A fill pauses the pipeline’s other tables while it runs, as a table’s copy does. On a large
table that is as long as reading the column takes. A plain
ADD COLUMNusually needs no fill (above). - On a PostgreSQL destination a fill is a chunked
UPDATE. The values are written into your table by key, a chunk at a time, each chunk committed on its own. YourUPDATEtriggers fire for every row it updates, and each such row leaves a dead tuple forVACUUMto clear, as any update does. - On a PostgreSQL destination the pipeline waits at most five seconds for a lock on your table.
Each change it makes to your table runs alone, in its own transaction, under
lock_timeout = '5s', so your other sessions never queue behind it for longer than that. Even a change made in place takes PostgreSQL’s strongest lock on the table (ACCESS EXCLUSIVE) while it runs. When another session holds the table, the pipeline tries four more times, 5, 10, 20 and 40 seconds apart. Then a table that is syncing stops, and nothing of it is written meanwhile: while the pipeline keeps running it tries again on its own every five minutes, and after a restart of the pipeline you resume it yourself. A table about to be copied waits instead, with the note that another session held its destination table (destination_busy), and is tried again every five minutes while the pipeline runs, and at its next start. To end the wait sooner, end the session that holds the table. - For a pipeline that existed before the change policy, the shape it has delivered is recorded at its first start after the release that brought the policy, from what your source and your destination table hold then. So a change made between the last place that pipeline saved in the change log and that first start is not told as a change; a column your source has and your destination table lacks is still added and its values copied. And a column dropped and added again under the same name and type in that window keeps, on the rows the source did not change afterwards, the values your destination held before. If you changed a table’s shape around that time, use Re-sync on it.
- A column added with a type that is dropped again before the pipeline reaches the change stops its table (a migration rolled back while the pipeline was stopped or behind, for example): the pipeline cannot tell what type the column needs at your destination, so the table stops, its place in the change log is let go, and resuming it copies it again from your source as it is now.
- A publication that does not publish truncates sends none, so a truncate at the source is not
mirrored, whatever A table is truncated says, and nothing tells you so on the table yet.
PostgreSQL publications send truncates unless their
publishsetting leavestruncateout; the publication statement the pipeline’s page shows sends them. To send them from one that does not, addtruncateto itspublishsetting and keep every kind of change it publishes now. The setting is the publication’s, so it applies to every table the publication lists: never addupdateordeleteto it on the way, unless every table it lists has a replica identity it can use, because once a publication sends updates and deletes your database refuses everyUPDATEandDELETEof a table it lists that has none (a table with no primary key and noREPLICA IDENTITY FULL, for one). - MySQL sources are not governed by the change policy. A change to a table’s shape at a MySQL source stops the pipeline, as before.