Exporting your account's history into your own destination
Your pipelines’ history is served for 30 days (run history) and kept in full for as long as the
retention rules allow. To keep it for longer, or to analyse it beside your own data, the
platform export serves it in a form a program can copy into your own warehouse, a row at a time
and without missing one: GET /v1/exports/{stream}, read with a read API key.
The history export reads this API for you: a source of kind platform, made from a read key,
and an ordinary pipeline per stream into a PostgreSQL database you own, once a day. Any other program
that pages through the API as described below gets every row too.
The six streams
Section titled “The six streams”| Stream | What a row is | Ordered by (the stream’s cursor) | Key |
|---|---|---|---|
pipelines |
a pipeline, deleted ones included (deleted_at set) |
id; read whole every time |
id |
events |
one row of a pipeline’s history, as GET /v1/pipelines/{id}/observations shows it |
sequence, per pipeline |
pipeline_id, sequence |
runs |
a run, or a session for a source that streams, as the run history shows it | changed_sequence (the run’s latest change), per pipeline, in no promised order |
pipeline_id, run_key |
usage_days |
a pipeline’s measured quantity for a UTC day, dimension and classification | day_number (days since 1970-01-01), per pipeline |
pipeline_id, day, dimension, classification |
wallet_entries |
one ledger entry: a debit, a refund, a top-up, a grant, an expiry | entry_id |
entry_id |
audit_events |
one audit row (owners’ keys only), never an address or a browser | occurred_at_us (microseconds) |
id |
events, runs and usage_days are read one pipeline at a time (pipeline_id is required):
their order holds within a pipeline, not across pipelines. Read pipelines first, then each
pipeline’s rows.
Exporting with pipelines
Section titled “Exporting with pipelines”On the Sources page, Export your history (linked from Usage and Settings › Audit log too, once the app knows the account’s API keys are on; where they are not, the page says so in place of its form, since the export is made from a read key) asks for three things and makes the rest:
- A read API key of this account (Settings, API keys, read access). It is checked before it is stored: a key of another account, a revoked or expired one, and a full key are refused (the export only reads). It is stored encrypted and never shown again.
- A PostgreSQL destination of this account. Snowflake is not offered for the export in this release.
- The streams. Each becomes a pipeline of its own, named
<export>-<stream>, because a pipeline from this source syncs one stream. Each runs every 24 hours unless you give it another schedule, and each is billed like any pipeline, for the resources its runs use (compute and memory); the rows and bytes it moves are measured so you can see them, and are not priced. A run that fails is free.
With the API: POST /v1/sources with {"id", "name", "kind": "platform", "api_key": "<read key>"},
then one POST /v1/pipelines per stream with source_id, destination_id and streams: ["<stream>"].
GET /v1/connectors lists the streams on the platform card.
Before the first sync, create each stream’s table in the destination’s public schema (the export’s page shows
these same statements, read from this page when the app is built). A
PostgreSQL destination is not created for you, and a missing table stops its pipeline, naming the
table. A table dropped after its pipeline wrote rows stops it too, at its next start: pipelines (read whole
at every sync) gets every row again from the next sync once you make the table again; for any other stream the
pipeline reads on from where it stopped, so to have every row again delete the pipeline, make the table again
and create a new export pipeline for the stream under a new id, which reads the export from its start. These are the statements, one per stream:
CREATE TABLE public.pipelines ( id text NOT NULL, source text NOT NULL, destination text NOT NULL, source_id text, destination_id text, update_method text NOT NULL, desired_state text NOT NULL, created_at timestamptz NOT NULL, updated_at timestamptz NOT NULL, deleted_at timestamptz, PRIMARY KEY (id));
CREATE TABLE public.events ( pipeline_id text NOT NULL, sequence bigint NOT NULL, kind text NOT NULL, run_id text, occurred_at timestamptz NOT NULL, detail jsonb NOT NULL, PRIMARY KEY (pipeline_id, sequence));
CREATE TABLE public.runs ( pipeline_id text NOT NULL, run_key text NOT NULL, changed_sequence bigint NOT NULL, shape text NOT NULL, run_id text, start_sequence bigint NOT NULL, end_sequence bigint NOT NULL, started_at timestamptz, ended_at timestamptz, outcome text, rows bigint, bytes bigint, detail text, last_confirmed_write timestamptz, no_write text, restarts bigint, retention_pressure text, PRIMARY KEY (pipeline_id, run_key));
CREATE TABLE public.usage_days ( pipeline_id text NOT NULL, day text NOT NULL, day_number bigint NOT NULL, dimension text NOT NULL, classification text NOT NULL, value bigint NOT NULL, PRIMARY KEY (pipeline_id, day, dimension, classification));
CREATE TABLE public.wallet_entries ( entry_id bigint NOT NULL, kind text NOT NULL, amount_minor bigint NOT NULL, occurred_at timestamptz NOT NULL, pipeline_id text, usage_day text, dimension text, classification text, run_id text, reason text, PRIMARY KEY (entry_id));
CREATE TABLE public.audit_events ( id text NOT NULL, occurred_at timestamptz NOT NULL, occurred_at_us bigint NOT NULL, action text NOT NULL, actor_kind text NOT NULL, actor_id text, actor_label text, via text NOT NULL, target_kind text, target_id text, target_label text, pipeline_id text, changes jsonb NOT NULL, reason text, request_id text, PRIMARY KEY (id));The pipelines keep their own marks (one per pipeline for the three per-pipeline streams), so there is
nothing to remember: each sync reads from where the last one stopped, re-reads the look-back windows
below, and upserts by key. Sync at least once every 30 days (the daily schedule does this for you):
runs that left the run history’s 30-day window before a sync read them are not served (their events
are), and events the history’s retention removed before a sync read them cannot be copied. When that
happens to the events pipeline, it stops and says so rather than step past them: see below.
When history was removed before the export read it
Section titled “When history was removed before the export read it”The history keeps every event for at least 31 days; after that, a run that has been over for 31 days
may lose its per-batch rows and measurements (batch_applied, resource_consumed,
retention_observed), as GET /v1/pipelines/{id}/observations says. An events pipeline that has not
synced for that long can find rows it never read already gone. It does not step past them: the export
refuses its read (410 history_removed), and the pipeline stops with a sentence naming the pipeline
whose history it was reading, how many rows were removed and which sequences. Nothing after them
is copied, no pipeline’s history is copied by it while it is stopped, and starting the pipeline
again stops it again, at this gap or at one it meets first. It names the first such pipeline only: it reads the pipelines in
order and stops at the first gap, so other pipelines’ histories it had not read yet may have lost
rows too, and the new pipeline below copies what is kept of each without naming those.
The rows cannot be brought back by anything. To go on from what is kept:
-
Make a new export pipeline for the
eventsstream, from the same source into the same table:POST /v1/pipelines{"id": "<a new id>", "source_id": "<the source>", "destination_id": "<the destination>", "streams": ["events"]}It has no mark yet, so it reads every pipeline’s history from what is kept, and writes over the rows the table already holds (the same keys, the same values). Rows your table copied before the history removed them stay in it: nothing is truncated or replaced. Its first sync reads every pipeline’s kept history again, and like any run it is billed for the compute and memory it uses (the rows and bytes it moves are measured, not priced). A re-sync is not offered for an export pipeline (
resync_not_supported: it has nothing to copy). -
Delete the stopped pipeline once the new one has synced.
A program reading the API directly goes on in one of two ways, both losing no row that is kept:
ask again with since set to the resume_since the refusal’s details names, the last removed
sequence (missing_through), or read from the beginning without since. Never with since set to
next_sequence: since is the last row you hold, and the export judges only the rows after it, so
a row you name there without holding it is never judged, and if the history removes it after the
refusal (it may, within minutes, while it catches up) you step past it without a word. Never from the history’s own mark (swept_before says when it reached, and is not a
place to resume): rows it keeps for good, a run’s start and end, its stops and errors, lie on both
sides of what it removed. A read without since is never refused. And a page of a resumed read
stops at a gap, with a next_cursor, so that the next page meets the refusal rather than stepping
past it. A next_cursor carries no judgement of its own beyond the read it belongs to: to come back
later, ask with since set to the last row you hold.
When the key stops working
Section titled “When the key stops working”A read key that is revoked, rolled or has expired is refused by the export (it answers 401), and each export pipeline stops with a sentence saying so. Two steps bring it back, and the second is not optional: replacing the key starts nothing.
-
Create a new READ key under Settings, API keys, and put it on the source:
POST /v1/credentials/{ref}/rotate{"value": "<the new read key>"}{ref}is the source’scredential_ref(GET /v1/sources/{id}). The new key is checked as the first one was: a full key, another account’s key or anything that is not a key of this platform is refused, and nothing is stored. -
Start each export pipeline again: Start on the pipeline’s page, or
POST /v1/pipelines/{id}/start. A stopped pipeline stays stopped until you do.
The app does not replace a platform source’s key yet; until it does, the call above is the way, or a new export made with the new key.
If the export answers 403 instead, the key is not at fault: API keys are not enabled for the account, and a new key would be refused too. Nothing on the source needs changing; once API keys are enabled for the account again, start each export pipeline again, as in step 2.
Reading it so that nothing is missed
Section titled “Reading it so that nothing is missed”- Rows come oldest first, except
runs. Page withnext_cursoruntil it isnull;limitis at most 200. Arunspage is one read of the run history: every run of it that changed at or after yoursince, in no promised order, however many (limitdoes not apply). Read every page before you save your mark. - Remember the greatest cursor value you saved, per stream, and per pipeline for the three per-pipeline streams.
- Next time, ask from that value with
since.sinceis inclusive, so the last row comes again: save rows by their key (an upsert), and it costs nothing. - Two streams want you to step back before you ask:
audit_events: one hour (3,600,000,000 microseconds) behind your saved value. A row is stamped when the change began, so a change that took a moment to save can arrive with a stamp earlier than one you already have. Stepping back an hour catches it.usage_days: seven days behind your saved value. A day’s figures are updated when a late measurement for it arrives. A correction more than seven days later reaches the day only when you read the stream again from the start.
runsserves a run again each time it changes, with a largerchanged_sequence: a row of its own, its own end, or a later give-up of the supervisor that ended it (a run whose process died staysrunninguntil then). Save the rows in ascendingchanged_sequence, upserting on the key, and you hold every run as it stands. A run longer than one read of the history comes in parts that share itsrun_key; the part with the greatestchanged_sequenceis the run as it stands, with its first row asstart_sequenceand its start time and volume null (never a smaller number).
What the export is, and is not
Section titled “What the export is, and is not”- It is a copy of what we keep, never a second calculation. Runs are the run history’s own runs; usage is the measured quantities; money is the ledger’s entries, never re-priced. Nothing we compute reads the export back.
- It serves what is still kept. Runs are served for the run history’s 30-day window (older runs
remain in
events). A gap in aneventspipeline’ssequenceis a row the history’s retention removed, never a row the export skipped, and a resumed read that would step past one it never read is refused instead (410 history_removed, above). The first export of an account copies what is kept while it reads: a row the history removes before that first read reaches it is not copied. - Reading it is billed like any pipeline, as the resources it uses (compute and memory): each export run is metered as its own run, and a failed run is free. Its rows and bytes are measured, never priced.