Skip to content

Exporting your account's history into your own destination

Your pipelines’ history is served for 30 days (run history) and kept in full for as long as the retention rules allow. To keep it for longer, or to analyse it beside your own data, the platform export serves it in a form a program can copy into your own warehouse, a row at a time and without missing one: GET /v1/exports/{stream}, read with a read API key.

The history export reads this API for you: a source of kind platform, made from a read key, and an ordinary pipeline per stream into a PostgreSQL database you own, once a day. Any other program that pages through the API as described below gets every row too.

Stream What a row is Ordered by (the stream’s cursor) Key
pipelines a pipeline, deleted ones included (deleted_at set) id; read whole every time id
events one row of a pipeline’s history, as GET /v1/pipelines/{id}/observations shows it sequence, per pipeline pipeline_id, sequence
runs a run, or a session for a source that streams, as the run history shows it changed_sequence (the run’s latest change), per pipeline, in no promised order pipeline_id, run_key
usage_days a pipeline’s measured quantity for a UTC day, dimension and classification day_number (days since 1970-01-01), per pipeline pipeline_id, day, dimension, classification
wallet_entries one ledger entry: a debit, a refund, a top-up, a grant, an expiry entry_id entry_id
audit_events one audit row (owners’ keys only), never an address or a browser occurred_at_us (microseconds) id

events, runs and usage_days are read one pipeline at a time (pipeline_id is required): their order holds within a pipeline, not across pipelines. Read pipelines first, then each pipeline’s rows.

On the Sources page, Export your history (linked from Usage and Settings › Audit log too, once the app knows the account’s API keys are on; where they are not, the page says so in place of its form, since the export is made from a read key) asks for three things and makes the rest:

  1. A read API key of this account (Settings, API keys, read access). It is checked before it is stored: a key of another account, a revoked or expired one, and a full key are refused (the export only reads). It is stored encrypted and never shown again.
  2. A PostgreSQL destination of this account. Snowflake is not offered for the export in this release.
  3. The streams. Each becomes a pipeline of its own, named <export>-<stream>, because a pipeline from this source syncs one stream. Each runs every 24 hours unless you give it another schedule, and each is billed like any pipeline, for the resources its runs use (compute and memory); the rows and bytes it moves are measured so you can see them, and are not priced. A run that fails is free.

With the API: POST /v1/sources with {"id", "name", "kind": "platform", "api_key": "<read key>"}, then one POST /v1/pipelines per stream with source_id, destination_id and streams: ["<stream>"]. GET /v1/connectors lists the streams on the platform card.

Before the first sync, create each stream’s table in the destination’s public schema (the export’s page shows these same statements, read from this page when the app is built). A PostgreSQL destination is not created for you, and a missing table stops its pipeline, naming the table. A table dropped after its pipeline wrote rows stops it too, at its next start: pipelines (read whole at every sync) gets every row again from the next sync once you make the table again; for any other stream the pipeline reads on from where it stopped, so to have every row again delete the pipeline, make the table again and create a new export pipeline for the stream under a new id, which reads the export from its start. These are the statements, one per stream:

CREATE TABLE public.pipelines (
id text NOT NULL,
source text NOT NULL,
destination text NOT NULL,
source_id text,
destination_id text,
update_method text NOT NULL,
desired_state text NOT NULL,
created_at timestamptz NOT NULL,
updated_at timestamptz NOT NULL,
deleted_at timestamptz,
PRIMARY KEY (id)
);
CREATE TABLE public.events (
pipeline_id text NOT NULL,
sequence bigint NOT NULL,
kind text NOT NULL,
run_id text,
occurred_at timestamptz NOT NULL,
detail jsonb NOT NULL,
PRIMARY KEY (pipeline_id, sequence)
);
CREATE TABLE public.runs (
pipeline_id text NOT NULL,
run_key text NOT NULL,
changed_sequence bigint NOT NULL,
shape text NOT NULL,
run_id text,
start_sequence bigint NOT NULL,
end_sequence bigint NOT NULL,
started_at timestamptz,
ended_at timestamptz,
outcome text,
rows bigint,
bytes bigint,
detail text,
last_confirmed_write timestamptz,
no_write text,
restarts bigint,
retention_pressure text,
PRIMARY KEY (pipeline_id, run_key)
);
CREATE TABLE public.usage_days (
pipeline_id text NOT NULL,
day text NOT NULL,
day_number bigint NOT NULL,
dimension text NOT NULL,
classification text NOT NULL,
value bigint NOT NULL,
PRIMARY KEY (pipeline_id, day, dimension, classification)
);
CREATE TABLE public.wallet_entries (
entry_id bigint NOT NULL,
kind text NOT NULL,
amount_minor bigint NOT NULL,
occurred_at timestamptz NOT NULL,
pipeline_id text,
usage_day text,
dimension text,
classification text,
run_id text,
reason text,
PRIMARY KEY (entry_id)
);
CREATE TABLE public.audit_events (
id text NOT NULL,
occurred_at timestamptz NOT NULL,
occurred_at_us bigint NOT NULL,
action text NOT NULL,
actor_kind text NOT NULL,
actor_id text,
actor_label text,
via text NOT NULL,
target_kind text,
target_id text,
target_label text,
pipeline_id text,
changes jsonb NOT NULL,
reason text,
request_id text,
PRIMARY KEY (id)
);

The pipelines keep their own marks (one per pipeline for the three per-pipeline streams), so there is nothing to remember: each sync reads from where the last one stopped, re-reads the look-back windows below, and upserts by key. Sync at least once every 30 days (the daily schedule does this for you): runs that left the run history’s 30-day window before a sync read them are not served (their events are), and events the history’s retention removed before a sync read them cannot be copied. When that happens to the events pipeline, it stops and says so rather than step past them: see below.

When history was removed before the export read it

Section titled “When history was removed before the export read it”

The history keeps every event for at least 31 days; after that, a run that has been over for 31 days may lose its per-batch rows and measurements (batch_applied, resource_consumed, retention_observed), as GET /v1/pipelines/{id}/observations says. An events pipeline that has not synced for that long can find rows it never read already gone. It does not step past them: the export refuses its read (410 history_removed), and the pipeline stops with a sentence naming the pipeline whose history it was reading, how many rows were removed and which sequences. Nothing after them is copied, no pipeline’s history is copied by it while it is stopped, and starting the pipeline again stops it again, at this gap or at one it meets first. It names the first such pipeline only: it reads the pipelines in order and stops at the first gap, so other pipelines’ histories it had not read yet may have lost rows too, and the new pipeline below copies what is kept of each without naming those.

The rows cannot be brought back by anything. To go on from what is kept:

  1. Make a new export pipeline for the events stream, from the same source into the same table:

    POST /v1/pipelines
    {"id": "<a new id>", "source_id": "<the source>", "destination_id": "<the destination>", "streams": ["events"]}

    It has no mark yet, so it reads every pipeline’s history from what is kept, and writes over the rows the table already holds (the same keys, the same values). Rows your table copied before the history removed them stay in it: nothing is truncated or replaced. Its first sync reads every pipeline’s kept history again, and like any run it is billed for the compute and memory it uses (the rows and bytes it moves are measured, not priced). A re-sync is not offered for an export pipeline (resync_not_supported: it has nothing to copy).

  2. Delete the stopped pipeline once the new one has synced.

A program reading the API directly goes on in one of two ways, both losing no row that is kept: ask again with since set to the resume_since the refusal’s details names, the last removed sequence (missing_through), or read from the beginning without since. Never with since set to next_sequence: since is the last row you hold, and the export judges only the rows after it, so a row you name there without holding it is never judged, and if the history removes it after the refusal (it may, within minutes, while it catches up) you step past it without a word. Never from the history’s own mark (swept_before says when it reached, and is not a place to resume): rows it keeps for good, a run’s start and end, its stops and errors, lie on both sides of what it removed. A read without since is never refused. And a page of a resumed read stops at a gap, with a next_cursor, so that the next page meets the refusal rather than stepping past it. A next_cursor carries no judgement of its own beyond the read it belongs to: to come back later, ask with since set to the last row you hold.

A read key that is revoked, rolled or has expired is refused by the export (it answers 401), and each export pipeline stops with a sentence saying so. Two steps bring it back, and the second is not optional: replacing the key starts nothing.

  1. Create a new READ key under Settings, API keys, and put it on the source:

    POST /v1/credentials/{ref}/rotate
    {"value": "<the new read key>"}

    {ref} is the source’s credential_ref (GET /v1/sources/{id}). The new key is checked as the first one was: a full key, another account’s key or anything that is not a key of this platform is refused, and nothing is stored.

  2. Start each export pipeline again: Start on the pipeline’s page, or POST /v1/pipelines/{id}/start. A stopped pipeline stays stopped until you do.

The app does not replace a platform source’s key yet; until it does, the call above is the way, or a new export made with the new key.

If the export answers 403 instead, the key is not at fault: API keys are not enabled for the account, and a new key would be refused too. Nothing on the source needs changing; once API keys are enabled for the account again, start each export pipeline again, as in step 2.

  1. Rows come oldest first, except runs. Page with next_cursor until it is null; limit is at most 200. A runs page is one read of the run history: every run of it that changed at or after your since, in no promised order, however many (limit does not apply). Read every page before you save your mark.
  2. Remember the greatest cursor value you saved, per stream, and per pipeline for the three per-pipeline streams.
  3. Next time, ask from that value with since. since is inclusive, so the last row comes again: save rows by their key (an upsert), and it costs nothing.
  4. Two streams want you to step back before you ask:
    • audit_events: one hour (3,600,000,000 microseconds) behind your saved value. A row is stamped when the change began, so a change that took a moment to save can arrive with a stamp earlier than one you already have. Stepping back an hour catches it.
    • usage_days: seven days behind your saved value. A day’s figures are updated when a late measurement for it arrives. A correction more than seven days later reaches the day only when you read the stream again from the start.
  5. runs serves a run again each time it changes, with a larger changed_sequence: a row of its own, its own end, or a later give-up of the supervisor that ended it (a run whose process died stays running until then). Save the rows in ascending changed_sequence, upserting on the key, and you hold every run as it stands. A run longer than one read of the history comes in parts that share its run_key; the part with the greatest changed_sequence is the run as it stands, with its first row as start_sequence and its start time and volume null (never a smaller number).
  • It is a copy of what we keep, never a second calculation. Runs are the run history’s own runs; usage is the measured quantities; money is the ledger’s entries, never re-priced. Nothing we compute reads the export back.
  • It serves what is still kept. Runs are served for the run history’s 30-day window (older runs remain in events). A gap in an events pipeline’s sequence is a row the history’s retention removed, never a row the export skipped, and a resumed read that would step past one it never read is refused instead (410 history_removed, above). The first export of an account copies what is kept while it reads: a row the history removes before that first read reaches it is not copied.
  • Reading it is billed like any pipeline, as the resources it uses (compute and memory): each export run is metered as its own run, and a failed run is free. Its rows and bytes are measured, never priced.