Skip to content

A pipeline's history, a run's own page, and the overview

What a pipeline did, who changed it, and how the account looks at a glance. Everything here is read from the account’s own records when you open it; nothing is a stored summary that could fall behind.

A pipeline’s History tab lists what happened to it, newest first:

  • its events: syncs, copies, stops, schema changes, measurements, each in its own words;
  • the changes people made (and API keys, and our operations team): who stopped it, changed its schedule, added a table, with the before and after the audit log keeps. A change shows no address or user agent; owners read those in Settings, Audit log.

Three controls narrow it:

  • Show: events and changes, only the changes people made, or only the pipeline’s own events.
  • Changed by: the changes of one person or key, among those on the page.
  • Show measurements: the source log’s pressure and the CPU and memory measurement arrive about once a minute. They are hidden unless you ask for them.

Download history saves the last day, 7 days or 30 days of the same view as CSV or NDJSON. The file is data: occurred_at (UTC), kind, sequence, run_id, actor, detail (the worker’s own words, or a change’s action) and payload (the fields the history shows, as JSON). One download holds at most 31 days and 200,000 rows. A CSV cell that a spreadsheet would run as a formula starts with '.

Each run in the run history has a Link: /pipelines/<pipeline>/runs/<run> shows that run as the history does, with its own events below it. Send it to a teammate or keep it; opened by someone who is not signed in, it asks them to sign in and then opens the run. Run history covers the last 30 days: a link to an older run says so, and the run’s events are in the pipeline’s History, as long as the history keeps them (below).

Every event is kept for at least 31 days. Once a run has been over for 31 days, its per-batch rows (batch_applied) and its measurements (resource_consumed, retention_observed) may be removed, except the pipeline’s newest of each kind. The rows, bytes, CPU and memory they recorded stay in Usage; the source log readings (retention_observed) are not kept elsewhere. Every other event (starts, stops, errors, schema changes, copies and changes people made) is kept.

“At least” and “may” are meant: a run that failed, or one still running, keeps its rows longer, and a row whose figures our meter has not recorded is kept for good. When anything of a pipeline has been removed, the History says from when (“Runs that ended before …”), and the API’s history page carries swept_before. A row’s sequence is never reused, so a gap in the sequence is a removed row. The run history is never affected: no run it can show has lost a row.

Where a page of the run history begins. A page reads up to 20,000 of the pipeline’s events, within the 30 days, and a run can begin before it. Such a run keeps its outcome: one that moved rows before the page and then failed is partial, never failed. It shows no start time and no row or byte count unless the page holds its start and every row it moved (null, never a smaller number), and a session shows when it last wrote even when that was before the page. A run that ended before the page is not shown on it (an older page shows it whole, within the 30 days), and nothing on the page ends it again.

After signing in you land on the Overview:

  • N pipelines: how many need attention, are out of credit, retrying, behind schedule, catching up, healthy or paused. Each count opens the list filtered to it.
  • Alerts open now, errors and warnings, and under the count the Open problems: the first five open alerts in the Alerts page’s order (errors first, the oldest first), each with its pipeline, then See all with the account’s own count, never the number of rows shown. The rows are one read of the open alerts (up to 200) when the page opens, never repeated; when the account has more open alerts than that and some of its errors are not among them, a line says so and sends you to the Alerts page. With nothing open, nothing is drawn under the count.
  • Credit: the balance, and whether pipelines are paused for credit.
  • This month: this month’s billed charges, what was refunded and the net, in the account’s currency, the same figures Billing shows (UTC month).
  • Deleted pipelines whose clean-up is not done: while there are any, one line counts them and opens the Pipelines list. Its Deleted pipelines section keeps each one, marked while its clean-up is not done; once you say you have dealt with it, the list marks it Nothing left to do and the Overview’s line no longer counts it.
  • Getting started: add a source and a destination, create and start a pipeline, write your first rows, choose where to be told when something goes wrong, invite a teammate. Each step is done while its fact holds; an owner can hide the list (in their own browser).

Two of the figures above in more detail, each with the GET /v1/overview field it is drawn from (every figure on the page is that one read’s, but the open problems’ rows):

  • Deleted pipelines still owing a clean-up on your source (pipelines.deleted_owing_source_cleanup): how many deleted pipelines left a replication slot and a publication that nobody has marked as dealt with on the deleted pipeline’s page (Deleting a pipeline). We never read your source to check, so the count says what your team has marked, not what your database holds.
  • This month so far (spend.month_to_date): what this UTC month’s usage has been charged, what was credited back for it as refunds of runs that did not finish, and the net: the Billing page’s figures for the month. Each counts in the month of the usage it is for, so a refund made today for a run that started last month counts partly in last month. On the statements, that part is last month’s while last month’s statement is open; once it has closed, it is on this month’s statement, marked late, with its usage day. They are charges, not an estimate, so they trail what is running: a UTC day’s charges count once that day is billed, at the first hourly check after it ends, or later while its usage is still being measured, as each check retries the last three closed days. Today’s estimate is on the Usage page.

The list filters by health, source and destination, and searches by name (any part of it, in any case). Source and Destination each offer the account’s sources or destinations by name, read once when the list opens: at most 200 of each, and where there are more, a line says only the most recently created are listed (open another and press Show in Pipelines on its page). A filter set by that link is kept as its own choice, named by its id, even when the list does not hold it. Each filter is the page’s address (?health=, ?source=, ?destination=), so a filtered list can be bookmarked or sent. Health is worked out when the list is read, so one request looks at up to 1,000 pipelines for a health filter; when it stops before filling the page it says “Showing the first matches. Load more to keep searching”, and Load more carries on.

Pin on a row keeps that pipeline in the sidebar’s Shortcuts, with Recent below the pins: up to five pipelines you opened recently, newest first. A pinned pipeline is never in Recent: pinning one takes it out, and no older pipeline takes its place; after you unpin it, it comes back the next time you open it. You can pin up to ten. Both lists are kept in this browser for you in this account (another account, or another person on this browser, sees their own). Sign out, and leaving an account with Leave, clear every Shortcuts list in this browser: every account’s and every person’s, not only yours. Nothing is sent to us. In the icon rail, Shortcuts is one menu behind a pin.

Route What
GET /v1/pipelines/{id}/observations?include=changes the history with the changes; kind= (repeatable), actor=, measurements=hide
GET /v1/pipelines/{id}/observations/export from, to, format=csv or ndjson, and the same filters
GET /v1/pipelines/{id}/runs/{run_id} one run as the run history lists it, with shape
GET /v1/overview the overview: pipelines.deleted_owing_source_cleanup, spend.month_to_date (month, starts_at, ends_at, charged_minor, refunded_minor, net_minor) and the rest
GET /v1/alerts?limit=200 the open alerts, as the Alerts page reads them: the Overview’s open problems are its first page
GET /v1/pipelines?health=&q=&source_id=&destination_id= the list’s filters; scan_budget_reached when a health search stopped short

A member and a read API key may read all of them. The API reference has every field and refusal.