Deleting a pipeline
Deleting a pipeline stops it for good. It is the way out of a pipeline you no longer want, and of one that has stopped and cannot carry on: a stopped pipeline still holds its destination table, and no other pipeline can write that table until the first one is deleted.
What happens to each thing the pipeline touches
Section titled “What happens to each thing the pipeline touches”| What | What a delete does |
|---|---|
| The pipeline | It stops. If it is running, it finishes the batch it is writing first. It cannot be started again. |
| Your destination | Nothing is dropped. The tables it wrote stay, with every row in them, a table you removed from it earlier included. They are your data. A pipeline that never ran wrote none. |
| Your source | Nothing is touched. We never change anything on a source. The replication slot and the publication you created for a PostgreSQL pipeline stay there, and you are given the statements that remove them (below). |
| Its source and destination, and their stored credentials | Kept in your account. Deleting a pipeline does not remove the source and the destination it used, or the credentials stored for them. There is no way to remove a stored source, destination or credential yet. To make a stored credential useless, revoke it where it was issued: change the database user’s password, or unset the Snowflake user’s public key. |
| Its history | Kept. The pipeline’s page stays readable, with its activity and its runs. |
| Its name | Retired. A new pipeline cannot take a deleted pipeline’s name. |
| Its destination table | Free for another pipeline, as soon as the deleted one has stopped. |
A delete cannot be undone.
In the app
Section titled “In the app”On the pipeline’s page, Settings › Danger zone has a Delete pipeline… button, which opens a step that names the pipeline and lists each consequence above (the header’s … menu, More actions, also has Delete pipeline…, which takes you to that pane with the step already open). You confirm by typing the pipeline’s name, exactly.
Afterwards the page shows the deleted pipeline: when it was deleted, who deleted it and the
reference of the request that did, what it can have left in your destination and on your
source, and its history. Pipelines lists deleted pipelines apart from live ones, and marks
one whose source clean-up nobody has marked as dealt with. The overview counts those
(GET /v1/overview’s pipelines.deleted_owing_source_cleanup).
The page says only what can have happened
Section titled “The page says only what can have happened”What a pipeline left behind depends on how far it got, and the page reads that from our own record of it:
| The pipeline | What the page says |
|---|---|
| never ran (no worker of it ever reported) | We have no record of it writing anything to your destination or reading your source. No table and no working schema is named. The source statements are still shown: you create the slot and the publication before the first run, so they can exist. |
| ran, and never saved progress | It may have created its tables and begun to fill them. The tables it would write are named. |
| saved progress (your destination confirmed a write) | The tables it wrote are named, and for Snowflake the working schema beside them. |
The confirm step follows the same rule: it names the tables that stay only for a pipeline that ran.
“Deleted, still stopping”
Section titled ““Deleted, still stopping””If the pipeline was running when you deleted it, its worker finishes the batch in flight and then stops. Until it has stopped, the page says Deleted, still stopping, the pipeline still writes its table, and no other pipeline can. This usually takes seconds. Check again refreshes the page.
What is left on your source, and how to remove it
Section titled “What is left on your source, and how to remove it”A PostgreSQL pipeline reads a replication slot and a publication on your database. You create both, before the pipeline’s first run; we never create or drop anything on a source. So whatever you created is still there after a delete, also when the pipeline never ran.
A replication slot that nothing reads holds your database’s write-ahead log until the disk fills. So drop it, unless you mean to use it for something else.
The deleted pipeline’s page shows the exact statements, with your slot’s and publication’s names filled in, and a button that copies them. They have this shape:
-- See what the slot is holding. Changes nothing.SELECT slot_name, active, wal_status, pg_catalog.pg_size_pretty(pg_catalog.pg_wal_lsn_diff(pg_catalog.pg_current_wal_lsn(), restart_lsn)) AS retained_walFROM pg_catalog.pg_replication_slots WHERE slot_name = '<slot>';
-- Drop the slot and the publication.SELECT pg_catalog.pg_drop_replication_slot(slot_name) FROM pg_catalog.pg_replication_slots WHERE slot_name = '<slot>';DROP PUBLICATION IF EXISTS <publication>;- Run them on your source database as a superuser, or as a role that has
REPLICATIONand owns the publication. - Each does nothing if its object is already gone, so they are safe to run twice.
- PostgreSQL refuses to drop a slot that something is still reading. If the page says Deleted, still stopping, wait until it says Deleted.
- A pipeline created before slot names were unique to a pipeline says so on its page:
another pipeline with a similar name may read the same slot. Run the check first, and drop
the slot only if
activeisfalseand no other pipeline uses it.
If you set any of its tables to REPLICA IDENTITY FULL for this pipeline
Section titled “If you set any of its tables to REPLICA IDENTITY FULL for this pipeline”The statement the setup shows for a table whose primary key is DEFERRABLE gives it
REPLICA IDENTITY FULL (or changes nothing, and says why, where PostgreSQL would refuse the table’s
UPDATE and DELETE even under it: a publication that sends them has a column list for it, or, on
PostgreSQL 18 or later, does not send one of its generated columns). For as long as it stays, every UPDATE and DELETE on that table writes the whole
old row to your write-ahead log. Once nothing else needs it, set it back, only after the
DROP PUBLICATION above, and never while any publication sends the table, by its own name or
through a partitioned table above it, another tool’s
included: with a DEFERRABLE primary key, DEFAULT is no replica identity, and PostgreSQL then
refuses every UPDATE and DELETE on a table that is published.
The deleted pipeline’s page gives one read-only query over every table the pipeline selected,
a table you removed from it included, and over each of their partitions (the API gives the same
query, as the deletion’s source_cleanup.identity_reset.check). We keep no record of which tables are
FULL, so the query asks your database. It lists only the tables and partitions that are FULL now,
none under DEFAULT or USING INDEX, each with the publications that still send it, by its own name
or through a table it is a partition of, and for each one no publication sends any more it prints
the statement that sets it back. Copy the query from the page: it is composed for your pipeline’s
tables, and this guide prints no copy of it, which could only fall behind the one the page shows.
Run it after the DROP PUBLICATION above, then run the set_back of each table you set FULL for
this pipeline, as printed. set_back is empty while any publication still sends the table. No row
means nothing to set back.
- Only if you changed it for us. If a table was
FULLbefore, or another pipeline or subscriber needs it, leave it. The page shows this query in a block of its own: the Copy statements button does not copy it. - Setting a table’s replica identity takes an
ACCESS EXCLUSIVElock on the table, which blocks every read and write of it while it waits and runs. Withlock_timeoutset first, in the same session, it gives up after five seconds instead of queueing behind your traffic: run it again later if it does.
When you have dropped them, or decided to keep them, press I have dealt with this. The reminder stops. We do not check your source: this records what you told us, who said it, and when.
A pipeline whose source leaves nothing behind says This pipeline left nothing on your source.
What is left in a Snowflake destination
Section titled “What is left in a Snowflake destination”The pipeline’s tables, as for every destination. A Snowflake pipeline that ran also leaves its
working schema, _REYANSH_<SCHEMA>, beside the schema it wrote. Another pipeline that writes
the same schema shares it, so drop it only when no pipeline writes that schema. The page names
it. A pipeline that never ran created neither, and the page names neither.
To replicate the same tables again
Section titled “To replicate the same tables again”Create a new pipeline under another name. The new-pipeline wizard’s Review and create step shows the schema and table it will write before you create it. Where the schema is named after the pipeline (the default for a Snowflake destination), that is a new schema. Otherwise it is the table the deleted pipeline wrote, with the rows already in it.
Through the API
Section titled “Through the API”| Request | What it does |
|---|---|
DELETE /v1/pipelines/{id} |
Deletes. No body. Requires an Idempotency-Key. Answers 200 with the pipeline once nothing of it runs, or 202 while its worker is still stopping. After a 202, poll GET /v1/pipelines/{id} until deletion.state is deleted. Do not repeat the DELETE to find out: with the same key it replays the first answer, the 202 included, for as long as the key is kept. With a new key it changes nothing and answers the pipeline as it is now, never an error. |
GET /v1/pipelines/{id} |
Still answers for a deleted pipeline. Its deletion object carries state (deleting or deleted), deleted_at, deleted_by (user_id, request_id, and email for a member of your account), source_cleanup with the statements (and identity_reset: every table it selected, and a check that lists which are REPLICA IDENTITY FULL and prints the statement that sets each back), and, only when a worker of the pipeline ever ran, kept_tables (every table it wrote, a removed one included) and kept_working_schema. Before a delete, written_tables is the same list as kept_tables. The pipeline’s activity says how far it got: worker_ran and progress_saved. |
GET /v1/pipelines?include_deleted=true |
The list leaves deleted pipelines out unless asked. |
POST /v1/pipelines/{id}/source-cleanup/acknowledge |
Records that you have dealt with what was left on the source, with who said so (source_cleanup.acknowledged_by). Requires an Idempotency-Key. |
Every other change to a deleted pipeline (start, stop, its schedule, its settings) answers
409 with the code pipeline_deleted. Creating a pipeline under a deleted pipeline’s name
answers 409 with the code pipeline_exists.
An Idempotency-Key belongs to one route: the method and the path. The same key sent to
another pipeline’s DELETE is a different request, and it deletes that pipeline. Use a new
key for each delete.
The API has no route that removes a stored source, destination or credential yet, so a delete leaves them, and so does a create that was refused part of the way: the credential it stored and the source it created stay in your account under the pipeline’s name.