A Snowflake destination
A destination is a place your pipelines write to. You set a Snowflake destination up once, and then any number of pipelines can write to it, each in a schema of its own.
Today a Snowflake destination takes pipelines from a PostgreSQL source.
Before you start
Section titled “Before you start”You need a user in Snowflake who can use the SECURITYADMIN and SYSADMIN roles, to run the
setup script once. We never sign in as that user. We sign in as a service user the script
creates, with a key pair: there is no password.
1. A key pair: we make one, or you bring your own
Section titled “1. A key pair: we make one, or you bring your own”The form’s The key pair offers two ways.
-
Generate a key for me. We make an RSA-2048 key pair when you save. The private key is made and stored sealed on our side and is never shown to anyone, you included. Once you have saved, the destination’s page shows the setup script with the key’s public half already in it. There is nothing to make on your own machine.
-
Use a private key I have. For example, with OpenSSL 3 on your own machine:
openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 aes-256-cbc -v2prf hmacWithSHA256 -out rsa_key.p8openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pubrsa_key.p8is the private key: you paste it into the form (or choose the file), with the passphrase the first command asked for.rsa_key.pubis the public key, which goes into the setup script and nowhere else.
The private keys the form takes
Section titled “The private keys the form takes”Any RSA key of 2048, 3072 or 4096 bits, PEM-encoded, as OpenSSL, LibreSSL (the openssl built
into macOS) or Java wrote it:
| The file’s first line | What it is | Taken |
|---|---|---|
BEGIN PRIVATE KEY |
PKCS#8, not encrypted | Yes. |
BEGIN ENCRYPTED PRIVATE KEY |
PKCS#8, encrypted | PBES2: PBKDF2 under HMAC-SHA1, -SHA224, -SHA256, -SHA384 or -SHA512, with AES-128, AES-192 or AES-256 in CBC mode, or three-key 3DES. Or one of the six PKCS#12 schemes: RC4 (128- or 40-bit), 3DES (three- or two-key) and RC2 (128- or 40-bit). |
BEGIN RSA PRIVATE KEY |
PKCS#1 | Plain, or under OpenSSL’s older PEM encryption (Proc-Type and DEK-Info lines): DES, 3DES, or AES-128, -192 or -256. |
When you save, an encrypted key is decrypted once, with your passphrase, and stored sealed. We do not keep the passphrase. A passphrase with accented or other non-ASCII letters works, whichever OpenSSL version encrypted the key.
A key the form cannot take is refused with what to do instead: re-export it with
openssl pkcs8 -topk8 -v2 aes-256-cbc, or let us generate one. That covers an encryption we do
not decrypt (scrypt, AES-GCM, the old PBES1 schemes), a key of another size or type, and a file
holding something other than one private key.
2. Fill in the form and run the setup script
Section titled “2. Fill in the form and run the setup script”In the app: Destinations, then Add a destination, then Add a Snowflake destination.
| Field | What to enter |
|---|---|
| Destination name | How this destination is listed. It cannot be changed later. |
| Account identifier | orgname-accountname, from Snowsight’s account details. Not the full host, and not a region locator such as xy12345.us-east-1. |
| User, Role, Warehouse, Database | The names the script will create, or ones you already have. Type them exactly as SHOW USERS, SHOW ROLES, SHOW WAREHOUSES and SHOW DATABASES list them: Snowflake matches these names case-sensitively. |
| Time Travel (days) | Optional. How many days Snowflake keeps earlier versions of the tables your pipelines write, 0 to 90; Snowflake bills what it keeps as storage. Leave it empty to keep your account’s own setting. See Time Travel. |
| The key pair | Generate a key for me, or Use a private key I have with the whole private key file, from its first line to its last, and its passphrase if it has one. |
As you type, the form fills in the setup script. The script never contains a private key.
- With a key we generate, save first: saving makes the key, and the page then shows the script with its public key in place. Run it in a Snowflake worksheet.
- With your own key, you can run the script before saving: copy it, replace the public-key
placeholder with the text of
rsa_key.pub(without its first and last lines), and run it. After you save, the page shows it again with the public key in place.
What the script does, line by line
Section titled “What the script does, line by line”USE ROLE SECURITYADMIN;CREATE ROLE IF NOT EXISTS "<role>";GRANT ROLE "<role>" TO ROLE SYSADMIN;CREATE USER IF NOT EXISTS "<user>" TYPE = SERVICE DEFAULT_ROLE = "<role>" DEFAULT_WAREHOUSE = "<warehouse>";ALTER USER "<user>" SET RSA_PUBLIC_KEY = '<public key>' DEFAULT_ROLE = "<role>" DEFAULT_WAREHOUSE = "<warehouse>";GRANT ROLE "<role>" TO USER "<user>";USE ROLE SYSADMIN;CREATE WAREHOUSE IF NOT EXISTS "<warehouse>" WAREHOUSE_SIZE = XSMALL AUTO_SUSPEND = 60 AUTO_RESUME = TRUE INITIALLY_SUSPENDED = TRUE;CREATE DATABASE IF NOT EXISTS "<database>";GRANT USAGE ON WAREHOUSE "<warehouse>" TO ROLE "<role>";GRANT USAGE, CREATE SCHEMA ON DATABASE "<database>" TO ROLE "<role>";| Line | Why it is needed |
|---|---|
CREATE ROLE, GRANT ROLE … TO ROLE SYSADMIN |
One role holds everything we are allowed to do, so you can see it and take it away in one place. Granting it to SYSADMIN lets your administrators see and manage what the role creates. |
CREATE USER … TYPE = SERVICE |
A service user cannot sign in with a password, only with the key pair. |
ALTER USER … SET RSA_PUBLIC_KEY … DEFAULT_ROLE … DEFAULT_WAREHOUSE |
Registers the public key. The role is made the user’s default because a session that names no role runs as the user’s default, and a new user’s default cannot create a schema. Repeated on ALTER so running the script again with a new key or role takes effect. |
GRANT ROLE … TO USER |
Lets the user use the role. |
CREATE WAREHOUSE, GRANT USAGE ON WAREHOUSE |
Every statement runs in this warehouse. It resumes itself when a statement arrives and suspends after 60 seconds idle. |
CREATE DATABASE, GRANT USAGE, CREATE SCHEMA ON DATABASE |
USAGE to see the database, CREATE SCHEMA because each pipeline creates the schemas it writes, on its first run. |
What the script deliberately does not grant
Section titled “What the script deliberately does not grant”- Nothing on any existing schema or table. The role owns the schemas and tables it creates, so it needs no grant on them, and it can reach nothing else you keep in that database.
- No schema is created. A destination has no schema of its own: every pipeline has its own, named when you create the pipeline (see Where a pipeline writes).
If a pipeline is to write a schema that already exists, grant the role USAGE and
CREATE TABLE on that schema yourself. PUBLIC is one: every Snowflake database has it, and a
pipeline that keeps its source’s schema name writes PUBLIC when that name is public.
A table someone else created needs SELECT, INSERT, UPDATE, DELETE granted on that table.
3. Save, and check the fingerprint
Section titled “3. Save, and check the fingerprint”Save destination stores the settings and the private key, or makes the key if you chose to. The private key is stored encrypted, is never shown, and no API response returns it: you see its fingerprint, and its public key for the setup script.
In Snowflake, DESC USER "<user>" shows RSA_PUBLIC_KEY_FP. It should be the fingerprint the
app shows. If it is not, the user has a different public key from the one this private key
pairs with.
4. Test the destination
Section titled “4. Test the destination”Open the destination from Destinations, then its Test pane, and press Run the test. The destination is tested on its own: the test asks for nothing, and no source is
needed. It stores nothing. With the API, send POST /v1/destinations/{id}/test (no body; an
Idempotency-Key, as on every change). The pipeline form’s test of a source and this destination
runs the same destination steps, with the same words.
The answer is a list of steps, each passed, passed with a warning, failed, no answer, skipped or not run: reaching Snowflake, securing the connection, signing in, the role’s permissions, and the pipeline’s table, which is skipped because no pipeline has picked one yet. The first step that failed is the one the answer names, with what to change. The permissions step reads “no answer” when Snowflake did not answer the permission test’s create: that is not a pass, and the test should be run again.
What it checks:
- The destination: that the key signs in, and that the role, the warehouse and the database can be used.
- That the role may create what a pipeline creates, by creating it, as Fivetran’s setup test
does. In the destination’s database it creates a schema named
_REYANSH_check_<time>_<random>(<time>is digits,<random>is eight characters from0-9anda-f), a tablePROBEin it and a temporary tablePROBE_TEMP, then drops the schema, which takes the table with it; the temporary table lasts only for the statement that made it. It also lists the grants on the database and on the role, as a second check.
None of this starts your warehouse.
The permission test’s answer is one of three:
- Passed: the role created the schema, the table and the temporary table.
- Refused: Snowflake refused one of them for want of a privilege. The whole test then says
no, in a sentence of ours that quotes Snowflake’s error code and says what to change (for
example, for code
003001, the grantGRANT CREATE SCHEMA ON DATABASE ... TO ROLE ...). - No answer: the test was not run, because the check had no time left for it, or Snowflake gave no answer about privileges, or not in time. That is not a pass. Test again.
Beside it, the test shows what the grants list. When the two disagree, it says so, and why they can. The test is what counts.
- The test passed, and the grants do not list the privilege. The listing does not follow a grant made to a user or through a database role, and a grant made in the last minute may not be listed yet.
- The grants list the privilege, and the test was refused. A listed grant is not a create
that works. The role may have no
USAGEon the database, which the listing does not read; the database may be shared to your account from another one, where no schema can be created; the refusal may have been of a table inside the new schema, which the listing does not read; or a grant made in the last minute may not be in effect yet, so test again in a minute. The answer names the one that applies.
What it does not check: the pipeline’s own schema and tables, which do not exist yet. A
pipeline’s first run creates them with the CREATE SCHEMA grant the test has just used. A
pipeline told to write a schema that already exists, such as PUBLIC, needs USAGE and
CREATE TABLE on that schema, which the test does not check: a missing grant there shows at
the pipeline’s first run, as a stop that names the schema.
When the destination says no, the test says at which step and why, in a sentence of ours that quotes Snowflake’s error code where it gave one.
A test schema left behind
Section titled “A test schema left behind”The test drops its schema before it answers. When it cannot confirm the drop, or finds a schema an earlier test left, the answer lists each one with the statement that drops it. A later test, as the same role and when it has the time, drops a test schema the role owns once it is an hour or more old by the time in its name: at most eight a test, oldest first. It never drops one another role owns, or one less than an hour old, which another test may be using; those it lists.
When the test ends with no answer (it did not finish in time, or something went wrong at our end), it may have been stopped between its create and its drop, and it cannot say whether its schema is still there. The app says so below the test. To look, run this in Snowflake, with your database’s name exactly as you typed it in the form:
SHOW SCHEMAS IN DATABASE "<database>" STARTS WITH '_REYANSH_check_';STARTS WITH compares by case, so it lists only names that begin _REYANSH_check_ with check
in lower case. Do not look with LIKE: it ignores case, and lists the schemas below too.
Drop a schema only if its whole name is _REYANSH_check_, then digits, an underscore and eight
characters from 0-9 and a-f, and only if no other test is running. Keep the double quotes,
or Snowflake reads the name in upper case:
DROP SCHEMA IF EXISTS "<database>"."<its name>";Never drop a schema whose name goes on in capitals after _REYANSH_, such as
_REYANSH_CHECKOUT: a pipeline names its working schema that way (see
Where a pipeline writes), and writes through it.
Changing the warehouse, role or user
Section titled “Changing the warehouse, role or user”A destination’s warehouse, role and user can be changed (PATCH /v1/destinations/{id}, or
Check and save under Change its settings in the destination’s Settings pane); its account and database cannot, because they decide where every
pipeline’s tables are: create a new destination for another one. A new user needs its key pair with
it. Before anything is saved, the new settings are checked against every table the destination’s
pipelines write: if the role cannot use a schema, write a table or own it, nothing is saved and the
answer lists each table and the statements an administrator runs to allow it. Once saved, every running
pipeline of the destination restarts onto it within a minute. An edit waits while a key rotation is
pending.
Rotating the key
Section titled “Rotating the key”Open the destination’s Key rotation pane (Rotate the key). A rotation never leaves a running pipeline
signing with a key Snowflake does not know. Snowflake keeps two public keys per user,
RSA_PUBLIC_KEY and RSA_PUBLIC_KEY_2, and accepts a sign-in with either: the new key goes into
the second key position first, the pipelines move to it, and only then is the old key retired.
With a key we generate (Generate a new key), in three steps:
-
Register the new key in the second key position. The page shows the new key’s fingerprint and the statements. First run
DESC USER "<user>"and check thatRSA_PUBLIC_KEY_2_FPis empty. If it shows the fingerprint of the key your pipelines use, an earlier rotation never ran its step 3, and setting the second key position now would replace that key: the page shows the two statements to run first. Then:ALTER USER "<user>" SET RSA_PUBLIC_KEY_2 = '<new public key>'; -
Test and switch. We sign in to Snowflake with the new key, on every Snowflake destination that uses this key. Only if Snowflake accepts it is the stored key switched, and every pipeline using it restarts with the new key. If Snowflake refuses it, nothing changes: the page quotes Snowflake’s words, and the current key keeps working. Until you switch, Generate a different key replaces the waiting one; register that one instead.
-
Retire the old key. The page counts the running pipelines that have not yet restarted onto the new key, and shows this step when there are none:
ALTER USER "<user>" SET RSA_PUBLIC_KEY = '<new public key>';ALTER USER "<user>" UNSET RSA_PUBLIC_KEY_2;In this order. The first moves the new key into the first key position while it is still in the second, so at no moment is it in neither (Snowflake accepts the same key in both positions). The second leaves the second key position free for your next rotation.
With your own key (Use a private key I have): run step 1 with your new key’s public key first, then save the new private key. It takes effect at once, and every pipeline using it restarts with it. Then run step 3 once the page says every running pipeline has restarted.
A stopped pipeline is not counted, and is not waited for: it signs with the new key when it
starts. A key waiting to be switched to is not listed among your credentials and cannot be named
by a destination or a pipeline; names ending in -next are kept for it.
Time Travel
Section titled “Time Travel”Snowflake keeps the earlier versions of a table for its Time Travel retention, so you can
query them with AT or BEFORE, or bring back a dropped table with UNDROP. A destination’s
Time Travel (days) sets that retention for the tables its pipelines write: 0 turns it off,
and empty, the default, leaves your account’s own setting alone. It is chosen when you create the
destination.
The days are billed. Snowflake charges storage for the earlier versions it keeps, during Time Travel and during Fail-safe after it: “Storage fees are incurred for maintaining historical data during both the Time Travel and Fail-safe periods”, calculated “for each 24-hour period”, on the part of a table that changed (Snowflake’s Storage costs for Time Travel and Fail-safe, read 2026-11-06 (IST)). A pipeline changes its tables at every sync, so more days keep more earlier versions, and cost more.
Each pipeline applies it every time it starts:
- In a schema the pipeline alone writes (the default naming,
<pipeline>_<source schema>): as the schema’s own setting. Every table in the schema follows it, those made later included. We never set a table’s own retention there, so the schema’s setting is the only one. - In a schema it shares (named after the source’s schema, which may also hold your own tables): on each table the pipeline writes, and only those. We never change the schema’s setting, so your own tables in it keep theirs.
A table’s retention may be changed only by its owner, and a schema’s by its owner or a role
granted MODIFY on it (Snowflake’s privileges reference). The role owns the schemas and tables a
pipeline creates; a schema you made yourself keeps its own setting unless you grant the role
MODIFY on it, and a table you made unless you grant the role OWNERSHIP of it.
When Snowflake refuses it, the pipeline writes on. Its history shows Time Travel not set, naming the schema or table and why, and the next start tries again. Snowflake refuses it when:
- your account keeps fewer days: a Standard edition account keeps at most 1 day (Enterprise and above, up to 90);
- the table or schema is transient, which keeps at most 1 day (see below);
- the role may not change the schema (it needs
OWNERSHIPorMODIFY) or the table (it needsOWNERSHIP).
When Snowflake keeps more days than you set, the history says so too. Once a pipeline has
set the days, it reads back what Snowflake keeps, and where that is more than you set, its history
says how many days are kept and for which schema or table, each time it starts. Two things make
Snowflake keep a table’s history longer than its own setting: a minimum retention an account
administrator set for the whole account (MIN_DATA_RETENTION_TIME_IN_DAYS; Snowflake keeps the
larger of it and your setting for every permanent table, and transient tables are not held to
it), and a Snowflake stream on the table, which keeps the history its offset still needs. The
pipeline cannot tell which it is, so it names both. It does not read the account’s parameter
itself: Snowflake shows it as 0 to a role that is not an administrator, whatever it is.
Snowflake also keeps every dropped or changed row for 7 more days of Fail-safe after Time Travel ends, which you cannot query and which is billed as storage. Time Travel and Fail-safe both keep a column’s earlier values: setting Time Travel to 0 shortens how long, and does not end Fail-safe.
Transient tables: no Fail-safe
Section titled “Transient tables: no Fail-safe”If you do not want Fail-safe’s storage for what a pipeline writes, make the database transient before you run the setup script:
CREATE TRANSIENT DATABASE "<database>";The script’s CREATE DATABASE IF NOT EXISTS then leaves it as it is, and every schema and table
created in a transient database is transient: it keeps at most 1 day of Time Travel and has no
Fail-safe. A Time Travel setting above 1 day is then refused, as above. A permanent database
cannot be made transient afterwards.
Clustering
Section titled “Clustering”A pipeline never sets a clustering key, and no option to make it set one is offered, as Fivetran sets none: a table it writes is created without one. This is measured, not assumed. We clustered copies of a 10^8-row table by its key (a random UUID key, and the pipeline’s own row id for a table with no primary key), by the key itself and by its first three characters, and synced the same changes into each as a pipeline does. Every clustered copy cost the warehouse more, 6% to 35% per sync, not less. A sync’s changed rows have keys spread over the whole table, so even a clustered table has every micro-partition read; and writing into a clustered table splits it into many small micro-partitions, which the next sync then reads. Snowflake’s clustering service itself billed little (the first ordering about 0.06 credit for that table, each sync’s changes millionths of a credit), so the cost is the warehouse’s. Snowflake recommends clustering for very large tables that are queried often and change rarely; a table a pipeline writes changes at every sync.
A clustering key you set yourself is left as it is. If your own queries are faster with a key, set one: no pipeline sets, changes or drops a clustering key, and Snowflake bills its upkeep to your account as it reorders the table after each sync. Two things to know:
- A change of a table’s sync mode that builds a new table (see the sync modes guide) builds it without a clustering key. Your key stays with the old table, which is kept under its new name, as a Snowflake stream on it does; set the key again on the new table if you want it.
- A change of a column’s type that Snowflake cannot make in place stops the table when that column is in your key. Such a change is applied by renaming (see When a source table’s shape changes), and Snowflake does not let a column that is part of a clustering key be renamed or dropped. The table’s history says the column is part of the table’s clustering key. To carry the change on:
ALTER TABLE "<database>"."<schema>"."<table>" DROP CLUSTERING KEY;naming the database too, so a worksheet set to another database (a development copy of this one) cannot drop that copy’s key instead. The table’s history gives the statement with every name filled in. Then resume the table, and set your key again once the change is done.
What clustering costs you is in your own account, by table and day:
SELECT DATE_TRUNC('day', START_TIME) AS day, TABLE_NAME, SUM(CREDITS_USED) AS credits FROM SNOWFLAKE.ACCOUNT_USAGE.AUTOMATIC_CLUSTERING_HISTORY WHERE DATABASE_NAME = '<database>' AND START_TIME >= DATEADD('day', -7, CURRENT_TIMESTAMP()) GROUP BY 1, 2 ORDER BY 1, 2;The view can be up to three hours behind, and a role other than ACCOUNTADMIN reads it once granted
GRANT DATABASE ROLE SNOWFLAKE.USAGE_VIEWER TO ROLE <role>.
What your schedule costs your warehouse
Section titled “What your schedule costs your warehouse”A pipeline from PostgreSQL into Snowflake reads its source continuously and holds each quiet batch
for as long as its schedule puts between two runs, so the warehouse wakes about once per run rather than
at every change. Waking it is most of what a pipeline costs you in Snowflake: each resume bills at least
60 seconds, then every second until AUTO_SUSPEND ends it.
The schedule editor shows what a schedule may cost your warehouse, beside what it costs here, when you
create a pipeline into a stored Snowflake destination and on the pipeline’s page. It reads your warehouse’s
own settings (its size, its generation, AUTO_SUSPEND, its clusters and your account’s region) with
SHOW WAREHOUSES and CURRENT_REGION(), which wake nothing, and shows:
- credits a day, in one of two shapes. For a pipeline that streams, “up to” so many credits a day; for a pipeline that runs on a schedule, the average a day over the coming runs beside what one run may cost (below). Never money: your credit price is your contract with Snowflake. It is the warehouse time a day at Snowflake’s published rate for your warehouse (its Service Consumption Table, dated in the estimate): Gen2 warehouses bill 1.35 times Gen1 on AWS and Google Cloud and 1.25 times on Azure; if your account’s cloud cannot be read, the figure spans both and says so. Compute credits only: cloud-services credits are not counted. A rate the table does not list shows hours, and no credits; a warehouse kind the estimate does not describe (an interactive warehouse, which bills an hour per resume) gets no figure at all.
- why it is “up to”. A pipeline that streams writes only when it has a change: a quiet source wakes the warehouse less, or not at all. So the figure assumes a change in every hold, and the lower figure is shown only as “if changes arrive at every hold”. And a wake that lands on a warehouse already running for other work costs you nothing more.
- how it is worked out. Each wake bills at least 60 seconds, then the write (assumed to take at most a
minute; a longer one costs more) and
AUTO_SUSPEND, plus up to 30 seconds because Snowflake checks for idleness about every 30 seconds. A wake never bills past the next one: a schedule whose hold is shorter thanAUTO_SUSPENDkeeps the warehouse up all day, and the estimate says 24 hours, never more. A warehouse whoseAUTO_SUSPENDis 0 or unset does not suspend on its own: the editor says only that. If it is running, the schedule adds almost nothing to it; if it is suspended, the pipeline’s first write starts it, and it will not stop on its own. - a pipeline that runs on a schedule (query-based capture) wakes the warehouse once a run, at the schedule’s real times over the year ahead, so a weekly or monthly schedule is counted as weekly or monthly. It bills each run on the day it runs, so the editor shows the average a day over that window beside what one run may cost if its writes take at most a minute (a weekly run is “about 0.03 credits a day on average: each run up to 0.19”; a large run written in several batches can wake the warehouse more than once). One whose next run is more than a year away says so. No figure above zero is shown as 0.
- writes your volume forces. A batch is written early once it holds 100,000 changes or 32 MiB, so a
busy pipeline wakes the warehouse more often than its schedule. For a pipeline that has run, the
estimate counts that from its own measured volume over the last seven days; for a new one it says
that volume is not counted yet. A pipeline also writes early when your PostgreSQL server keeps a lot
of log for it: more than 1 GiB when
max_slot_wal_keep_sizeis not set, or half of the room left when it is (that log may be for other tables’ changes); those wakes are in no figure, and the editor says so.
Clusters: the estimate bills the clusters a resume starts (MIN_CLUSTER_COUNT). A warehouse that may start
more (MAX_CLUSTER_COUNT above it) may cost more than the figure, and the editor says so.
The settings are read again at most every ten minutes per destination (a read that failed, after a minute;
an edit of the destination, at once), and the read shares the connection test’s limit of six a minute; past
it the editor says the estimate is not shown, rather than guessing. To see what Snowflake actually metered,
query WAREHOUSE_METERING_HISTORY as above.
Where a pipeline writes
Section titled “Where a pipeline writes”Each pipeline writes its own schema in the destination’s database. You choose how the schema is named when you create the pipeline, and it cannot be changed afterwards.
| Naming | The schema | Example: pipeline orders, source schema public |
|---|---|---|
| Pipeline and source schema (default) | <pipeline name>_<source schema>, lower-cased, with every character outside a-z, 0-9 and _ replaced by _ |
ANALYTICS.ORDERS_PUBLIC |
| Source schema | the source’s schema name, exactly | ANALYTICS.PUBLIC |
The new-pipeline wizard’s Review and create step shows the exact schema and table before you create the pipeline. If another pipeline already writes that table, it says which one, and does not let you create a second writer for it.
-
Give every destination on one Snowflake account the same account identifier,
orgname-accountname. The check compares accounts by the identifier as written (case, and_against-, aside). An account given as its legacy locator (xy12345) is counted as the same account as everyorgname-accountnamenaming the same database, because which locator is which account only Snowflake knows: so a second pipeline over a table through the other spelling is refused, and so is a pipeline into a table of the same database, schema and name in another account, when one of the two destinations uses a locator. The refusal says so, and says what you can do if they really are two accounts. A pipeline’s destination is fixed when the pipeline is created, and a destination’s account cannot be changed, so:- creating a pipeline whose destination uses the locator: create it through a destination
that names the account as
orgname-accountname(SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME()shows it); - creating a pipeline when the OTHER pipeline’s destination uses the locator: create it
through a destination that names its own account by its locator (
SELECT CURRENT_ACCOUNT()shows it; a locator on its own works only for an account in AWS US West (Oregon)), because two different locators are two accounts; or delete the other pipeline and, once its worker has stopped (a create straight after is told it is still stopping), replace it with a new one through a destination that names its account asorgname-accountname(a deleted pipeline’s id is not reused, so the new one has another id); - adding a table to a pipeline that already exists: create a new pipeline for that table, through the destination the two cases above describe;
- starting a pipeline that already exists: it still counts as the table’s writer, so delete it first (its destination tables are kept, and it stops counting once its worker has stopped), then create a new pipeline for its tables (all of them, not only the refused one) through the destination the first two cases describe.
- creating a pipeline whose destination uses the locator: create it through a destination
that names the account as
-
Pipelines in two different workspaces are never compared: a workspace is never told what another one writes. Two pipelines writing one table that way both write it, every batch, and overwrite each other’s rows, and nothing refuses that: write one table from one workspace.
-
Schema and table names are written in upper case in Snowflake.
-
The pipeline also creates one working schema beside its own,
_REYANSH_<SCHEMA>(for the example above,ANALYTICS._REYANSH_ORDERS_PUBLIC). It holds the tables rows are staged in before they are merged, and a small table that stops two runs of one pipeline writing at once. Do not write to it. -
If the working schema, or its
_REYANSH_FENCEtable, is dropped while the pipeline’s tables hold rows, the next start waits up to about an hour before it writes. An earlier run’s statements can still be queued in Snowflake, and the record of who wrote last went with the table, so the start waits until any such statement has ended (at most 3,630 seconds), then carries on by itself. Its history says Waiting for earlier statements to end, with the minutes and what Snowflake’s own record showed, as the wait begins; the same row appears when it waits for another Snowflake user’s statements, after the destination’s user was changed. A new pipeline over tables that already hold rows waits once, on its first start, for the same reason, and its history says so: this includes a pipeline you make over the tables of one you deleted, whose last statements can still be queued in Snowflake for a while after it stops, and a table you add to a running pipeline when its destination table already holds rows another writer put there (the whole pipeline waits, once, at the start that adds it). The wait is per target schema: a pipeline whose tables are in several schemas, each owing a wait, waits for each in turn, about an hour apiece, telling each as it begins. An empty destination table counts as written while the working schema holds rows an earlier writer staged for it (in its staging table, or in the copy stages a first load of a table with no primary key uses): that writer’s first load may still be queued in Snowflake. A pipeline that makes its own tables, or finds them empty with nothing staged, does not wait. -
A name beginning
_reyansh_orpg_, andinformation_schema, are refused.
Large batches are loaded from files
Section titled “Large batches are loaded from files”A large batch (a first copy, or a busy stream) is written as compressed, encrypted files in the
stage of the user you gave us, under reyansh/, loaded with COPY, checked value for value
against what we sent, and merged into your table. It needs no grant: every user has its own
stage. The files are removed as soon as they are loaded. Small batches are written by INSERT.
The stage lives in the storage of the cloud that hosts your Snowflake account, and the files go straight there; nothing passes through storage of ours. On AWS, Google Cloud and Azure this route has been run and checked against a live Snowflake account on that cloud, every table compared value for value with the same rows written by INSERT. If Snowflake or its storage ever answers differently, the batch is written by INSERT instead, every row exactly as before, and the pipeline’s history says so (below).
How the upload signs in, for your security review. To upload, the pipeline signs in as the
user you gave us with the same key pair, through the sign-in Snowflake’s own drivers use, and it
names itself as Snowflake’s Go driver: Snowflake on Google Cloud gives a stage only to a driver it
knows. So in LOGIN_HISTORY these sign-ins show REPORTED_CLIENT_TYPE as the Go driver, with its
version; our own name is the session’s application, in CLIENT_ENVIRONMENT in
SNOWFLAKE.ACCOUNT_USAGE.SESSIONS. They run no query of yours and wake no warehouse.
If you run Reyansh yourself (your own cloud or servers), the machines that run pipelines must be
able to reach the stage’s storage over HTTPS: *.s3.<region>.amazonaws.com for an account on AWS,
*.googleapis.com on Google Cloud, *.blob.core.windows.net on Azure. If they cannot, large
batches are written by INSERT.
If the files cannot be used, large batches are written by INSERT instead: every row exactly as before, at more warehouse time. The pipeline’s history says so once, as Large batches written the slower way, with the reason and for how long:
- for the next 30 minutes, then files are tried again, when two large batches in a row could not use them: the stage’s storage refused or did not answer the upload, Snowflake did not answer or refused it with an HTTP 429 or 5xx, or Snowflake did not load the files as sent;
- until the pipeline next restarts, when Snowflake refused the upload’s sign-in or its request for the stage’s location in a way that does not pass by itself;
- for this account, when its stage uses a storage service, endpoint, bucket, region, storage account, container, encryption cipher or key this version does not upload to. A restart does not change that.
Files are not removed for a pipeline you delete, a table you deselect, or a service user you
replace. They are encrypted and at most a batch’s worth; remove them with REMOVE @~/reyansh/
as that user if you wish.
The columns we add
Section titled “The columns we add”Every table we create has your source’s columns, and three of ours (a table with no primary key has more of ours, its row’s hash and count among them: see A table without a key):
| Column | Type | What it holds |
|---|---|---|
_REYANSH_SYNCED_AT |
TIMESTAMP_TZ |
When we last wrote the row. |
_REYANSH_DELETED_AT |
TIMESTAMP_TZ, usually NULL |
When the row was marked deleted: deleted or truncated in your source, replaced by a change in your source (its key changed, or, in a table with no primary key, any of its values: the row as it was), found no longer in your source by a copy, or moved out of the table’s row filter by a change or a new filter (it is still in your source then). NULL means it is not marked. |
_REYANSH_META |
VARIANT, usually NULL |
A note for each value we could not write exactly as your source holds it: which column, why, and the original as text (for a value longer than its column holds, or an original too long to fit this column beside its rewritten value, its length instead). |
Deletes are soft by default
Section titled “Deletes are soft by default”For a table kept in soft delete, the default, a row deleted in your source stays in Snowflake and
_REYANSH_DELETED_AT is set. A truncate in your source marks every row of a table with no primary key whatever the truncate rule says, and does what the rule says to any other table. To read only the rows not marked deleted:
SELECT * FROM ANALYTICS.ORDERS_PUBLIC.ORDERS WHERE _REYANSH_DELETED_AT IS NULL;A row inserted again under the same key is live again: its _REYANSH_DELETED_AT goes back to
NULL. Soft delete is a table’s default mode; a table can be kept live instead, where a row deleted
in your source is deleted here (a query-based pipeline finds no row deleted in your source, so it stays here until
the table is copied again: see Query-based capture), and the rows marked deleted can be
purged, or are removed by the table’s next copy: see
Re-sync, Sync now, and a table’s sync mode.
The type each column lands in
Section titled “The type each column lands in”| Your PostgreSQL column | The Snowflake column |
|---|---|
boolean |
BOOLEAN |
smallint, integer, bigint, oid |
NUMBER(38,0) |
numeric(p,s), with p at most 38 and s from 0 to p (at most 37) |
NUMBER(p,s) |
numeric with no precision, or wider than 38 digits, or with a negative scale |
VARCHAR: the number’s exact text, never rounded. A destination may choose NUMBER(38,s) for these instead (below); a key column, and every column of a table synced without a key, stays VARCHAR |
real, double precision |
FLOAT |
text, varchar, char, name, citext, uuid, an enum |
VARCHAR |
bytea |
BINARY |
date |
DATE |
time |
TIME(9) |
timestamp |
TIMESTAMP_NTZ(9) |
timestamptz |
TIMESTAMP_TZ(9) |
json, jsonb, hstore, an array, a range |
VARIANT |
an array of one of the types below (money[], an array of a composite, …) |
VARIANT: JSON that keeps the array and its nesting, each element the value as PostgreSQL writes it |
interval |
VARCHAR: 1 year 2 mons 3 days 04:05:06.789012, months, days and seconds kept apart |
money |
VARCHAR: formatted by the lc_monetary the pipeline recorded from your database when it first started, so the same amount is $1,234.11 under en_US and 1.234,11 € under de_DE |
timetz |
VARCHAR: 12:34:56.789012+05:30, as TIME holds no offset |
inet, cidr, macaddr, macaddr8 |
VARCHAR: 192.168.0.1/24, 2001:db8::/32, 08:00:2b:01:02:03 |
point, line, lseg, box, path, polygon, circle |
VARCHAR: (1.5,-2.5) for a point |
PostGIS geometry, geography |
VARCHAR: hex EWKB, the shape and its SRID exactly (0101000020E6100000…) |
xml |
VARCHAR. PostgreSQL leaves a document’s <?xml version="1.0"?> declaration out when it sends the value; one that says standalone is kept |
| a composite | VARCHAR: its row value, (12.34,INR) |
anything else (bit strings, text search, ltree, pg_lsn, …) |
VARCHAR: the value as PostgreSQL writes it |
The pipeline’s page says, for each column of a table, which of these it is: open the table’s columns and read the line under each column. A column kept as text says why, by its kind.
A numeric column with no precision is common (amount numeric), and it lands as text,
so that no digit is lost: TO_NUMBER or TRY_TO_NUMBER reads it back as a number in a query.
Its NaN, Infinity and -Infinity are kept as that text. A declared numeric(p,s) holds 38 digits
at most here: numeric(38,0) and numeric(38,37) are NUMBER, numeric(39,0) is text, and so is a
negative scale (numeric(5,-2)), a scale above the precision (numeric(3,5)), or a scale of 38
(numeric(38,38): NUMBER declares a scale of 37 at most).
A char(n) value arrives padded to n characters, exactly as PostgreSQL sends it, and Snowflake
compares trailing spaces as part of the value: WHERE code = 'AB' does not match 'AB '.
Compare with RTRIM(code), declare the column varchar in your source, or choose to trim text
values (below). Unless you choose that, we never trim a value: what you see
is what your source holds.
Wide numerics as numbers
Section titled “Wide numerics as numbers”A destination can choose, when it is created, to land a numeric with no precision (or one wider
than 38 digits) as NUMBER(38,s) instead of text, for a scale s from 0 to 37: the form’s
Wide numerics land as (settings.wide_numeric_as, "NUMBER(38,9)" for example). Every account
may choose it, and it is chosen at create: it cannot be changed afterwards yet.
NUMBER(38,s)holds up to38 − sdigits before the point andsafter it. A value with more digits is written asNULL, never rounded: its exact text is kept in the row’s_REYANSH_METAunderdoes_not_fit, and the pipeline’s history says so (“Values changed when written”, naming the column and the rows), with a warning alert. That is how a table synced by a key (below) is written, as its changes stream and as it is copied. From scale 20, a 19-digit value such as a 64-bit id does not fit.- When the pipeline copies the existing values of a column added to such a table (a fill), a
value with more digits stops the table instead, held as
fill_failedand naming the column: nothing is written asNULL, because a fill cannot tell such a value. Resuming the table fills the column again, and stops again while your source holds such a value; re-syncing the table copies every row, and writes such a value asNULLand tells it, as above. - A table synced without a key keeps these columns as exact text,
VARCHAR, exactly as without the setting: its rows are written in a way that cannot tell a value written asNULL. Such a table is one whose rows are each identified by a hash of all its values, as the new-pipeline form marks it (A table without a key). A table the form marks with its primary key or with a unique index is synced by a key, and its columns take the setting as above. - A key column is always exact text, whatever the setting: a key is never written as
NULL, so a number setting would let one wide key value stop its table. - A declared
numeric(p,s)isNUMBER(p,s)whatever the setting says. - A column the table already holds as a narrower
NUMBER(p,t)(a type change your pipeline’s rules kept) is written as that type, exactly as without the setting: a value with more digits than it holds stops the table that holds it (below), and is never written asNULLfor it. Only the setting’s ownNUMBER(38,s)writes such a value asNULL, and only in a table synced by a key, as above. - A table this destination finds already holding such a column as text (adopted from another pipeline, or a column a schema change turned into text) keeps writing it as text, exactly.
To read a column with every value back, the NULLs included, take the original where the number is
NULL. The note is keyed by the column’s Snowflake name, in upper case (unless it is one of the six
names below):
SELECT COALESCE(TO_VARCHAR("AMT"), "_REYANSH_META":"AMT":"original"::VARCHAR) AS "AMT_EXACT"FROM <schema>.<table>;-- the rows the setting could not hold:SELECT * FROM <schema>.<table> WHERE "_REYANSH_META":"AMT":"unrepresentable" = 'does_not_fit';If a column the setting writes as a number, or may have, later becomes part of the table’s key (a
schema change turned it into text, then the source made it part of its primary key), its old rows may
hold Snowflake’s spelling of each number, or NULL, which a key compared as text never matches. The
pipeline records such a column before it changes or writes it, so a change that did not finish (a
refused step, a restart) can leave a column recorded whose rows were never numbers; either way the table
then stops before writing anything, and its message says how to go on: rename the table aside
(ALTER TABLE <schema>.<table> RENAME TO <schema>.<table>_BEFORE_KEY_CHANGE), then resume it, and it
is copied again into a new table, every row and its key written afresh. The renamed table keeps every
row, its _REYANSH_META notes, its grants and its Time Travel history, to read or drop when you choose.
Dropping the table instead also works, and loses what a copy cannot bring back: rows deleted at the
source and kept marked deleted, rows no longer in the source, a history table’s earlier versions and
an append-only table’s change rows, their notes, and the table’s grants and Time Travel history.
That stop relies on the pipeline that wrote the numbers. Which columns the setting wrote as numbers is remembered by the pipeline that wrote them, not by the table. If such a column was turned into text and the table is then written by another pipeline (one created again over the same tables, for example), that pipeline does not know the column’s old rows were numbers. It still stops the table when the column becomes part of the key if its own destination writes wide numerics as numbers and, at one of its starts before that, the column was a wide numeric at the source and not part of the key: each start records every column it would write as a number before it writes anything. Otherwise the table is not stopped first, and copying it again can leave rows that match none of the old ones, so a row can appear twice. Before you point a new pipeline at tables a wide-numeric destination wrote, rename those tables aside, as above.
Trimming text values
Section titled “Trimming text values”A destination can choose, when it is created, to remove the spaces and tabs at the start and end of
each text value before it is written: the form’s Trim spaces and tabs from text values
(settings.trim_whitespace, true). It is Airbyte’s Trim Whitespace from String Fields, made a
choice you make: Airbyte trims by default, and Fivetran’s Snowflake destination has no such
setting. Off is the default, and off, every text value lands byte for byte. Every account may
choose it.
- What is removed is exactly what Snowflake’s own
TRIM_SPACEremoves: the space and the tab, as many as stand at each end. We recorded it: Snowflake kept a no-break space, an ideographic space, an em space, a vertical tab and a form feed, and so do we, with a line break and the spaces between words. So' ab cd\t'lands as'ab cd', and achar(8)value'AB 'lands as'AB'. - Every value of the column is trimmed alike. Airbyte’s trim happens to the fields of its load file that it leaves unquoted, so a value holding a comma or a quote keeps its spaces there; here the rule is the value’s.
- A value of only spaces and tabs becomes the empty string, never
NULL. (Airbyte’s load turns such a value intoNULL.) - A key column is never trimmed:
'AB'and'AB 'are two keys, and trimmed they would be written as one row. Achar(n)key keeps its padding. - Only string columns are trimmed:
text,varchar,char(n),name,citext,uuidand enum columns. A value kept as your source’s text for another reason (anumerictoo wide for aNUMBER,money,interval,xml, a composite type) is kept exactly, and so is every JSON value, array, binary value and typed column such as a number or a date. A trimmed column says so in the pipeline’s column view (“trimmed”), where any other string says it lands exactly. - A column your source holds as another type and an existing table holds as text (the column view says it “lands in your table’s own type, converted”) is not a string either, and is kept exactly.
- It applies to every table the destination’s pipelines write, synced by a key or without one, as their changes stream and as they are copied. A table synced without a key still identifies each row by its values as your source holds them, so two source rows that differ only by a space stay two rows.
- A table kept in history, or as a log of changes, is written trimmed too, each version and each row alike, and its key exactly. In history, an update at your source that changes only the spaces at a value’s ends still writes a version, because a table kept in history has one version for each change your source made; it holds the same text as the version before, as an update that sets a value to itself does. A re-sync never writes such a version: it compares what it reads, trimmed, with the current version, finds the row unchanged and only updates its synced time.
- If a trimmed column later becomes part of a table’s key, its values are written untrimmed from then on, because a key never is. Rows written before keep their trimmed text: under soft delete, the trimmed row stays beside the new one, marked deleted, so the same source row can appear twice with and without its spaces. Re-sync the table to write it again from your source.
- A column you hash is hashed from your source’s value, spaces included: the trim applies to text written as it is, and a hash has no spaces to remove.
- It is chosen at create and cannot be changed afterwards: rows already written would keep their spaces while new ones lost them. To change it, create another destination and re-sync.
Column names
Section titled “Column names”Your columns’ names are written in upper case, as Snowflake writes a name that is not quoted:
a source column customer_id lands as CUSTOMER_ID.
Six names are the exception, and arrive in lower case. Snowflake refuses them as column names in upper case, even quoted, because the SQL standard reserves them:
current_date, current_time, current_timestamp, current_user, localtime, localtimestamp
A source column with one of these names, in whatever case your source writes it, lands under that name in lower case, so a query names it quoted, in lower case:
SELECT "localtime" FROM ANALYTICS.ORDERS_PUBLIC.EVENTS;CONSTRAINT, which Snowflake accepts quoted in upper case, stays upper case, like every other
column.
Values Snowflake cannot hold
Section titled “Values Snowflake cannot hold”Snowflake has no way to store a few values PostgreSQL can: an infinite date or timestamp, the
time 24:00:00, a timestamp before year 1, a numeric that is NaN or infinite, and a date
before year 1 or after the year 294276 in a column the destination holds as a timestamp (after a
schema change turned a source timestamp column into a date, and the column was kept: the
change’s line in the pipeline’s history names those dates). Such a value is written as NULL, and the
row’s _REYANSH_META keeps the original as text, so nothing is lost quietly and the pipeline goes
on. The pipeline’s history says so (“Values changed when written”), naming each column, how many
rows, and why, and a warning alert is raised. In a table kept in history or append the note is kept
in the row, but the pipeline’s history does not report it yet, and no alert is raised.
The exception is a numeric column that lands as VARCHAR (above): it holds text, so its
NaN is written as the text NaN, exactly as your source holds it, with no note in
_REYANSH_META.
A number with more digits than the NUMBER(38,s) a destination’s wide-numeric setting chose
(above) is written as NULL in the same way, its exact text kept under
{"original": ..., "unrepresentable": "does_not_fit"}, and told in the history, in a table synced by
a key as its changes stream and as it is copied. Copying an added column’s existing values (a
fill) stops the table at such a value instead (fill_failed), and a table synced without a key holds
these columns as text, so neither writes one as NULL.
A value longer than its column holds. A VARCHAR the destination creates holds 16,777,216
characters, whatever their width, and a BINARY 8,388,608 bytes. A longer text is cut to its
first 16,777,216 characters, as Fivetran cuts one; a longer binary, and a JSON value whose text is
longer than 16,777,216 characters, is written as NULL. Either way the row’s _REYANSH_META
says so ({"too_long": "cut", "characters": ..., "kept": 16777216}, or {"too_long": "nulled", ...}) with the value’s length, but not the original, which is too long for it as well; the
pipeline goes on, and its history and a warning alert say which column and how many rows.
Two things are never changed, and stop the table that holds them instead (the pipeline’s other tables keep syncing, unless When one table stops says to stop every table; the way back is in Values your destination cannot take):
- A value in a key column, too long ones included. A changed key would be a different row.
- A number with more digits than its
NUMBER(p,s)column holds, where the column is a declarednumeric(p,s), or a narrowerNUMBERa type change kept (a type nobody chose for it to overflow). It is never rounded. A column the wide-numeric setting made aNUMBER(38,s)writes such a value asNULLinstead, in a table synced by a key, and a fill stops the table at it, as above.
The “Values your destination cannot take” setting is read by PostgreSQL destinations only; see Values your destination cannot take.
Blocking, hashing and filtering
Section titled “Blocking, hashing and filtering”A Postgres to Snowflake pipeline can block a column, hash it, or filter a table’s rows: Blocking, hashing and filtering what a pipeline syncs has the whole of it. What is particular to Snowflake:
- A hashed column is
VARCHAR. Hashing a column of another type clears it, then drops it and adds it again asVARCHAR, so it moves to the end of the table. Unhashing returns it to its own type. - Blocking a column that holds rows can set it to NULL (the default), drop it, or keep it.
- A cleared value stays in Time Travel and Fail-safe until they end (see Time Travel): set Time Travel to 0 days before you block or hash a column if it must be gone sooner.
- Rows a new filter no longer matches are marked deleted (
_REYANSH_DELETED_AT) in a table kept in soft delete, removed in one kept live. - How each column is kept is also recorded in your warehouse, in
_REYANSH_SHAPINGin the pipeline’s_REYANSH_<SCHEMA>schema, which the pipeline compares at every start: a restore of either side stops the table rather than writing a column in clear.
The schedule
Section titled “The schedule”A Postgres to Snowflake pipeline reads your source continuously. Its schedule decides how long it holds changes before writing them to Snowflake, so your warehouse is resumed once per write and not at every change.
- Every 15 minutes, every hour, …: changes are written about that often.
- A cron expression with a time zone: changes are held for the shortest gap between two of the expression’s times in the coming week, to at most 24 hours.
- Manual: changes are held for 6 hours.
The app shows, for the schedule you pick, how long changes are held. It does not show clock times for “the next run”, because a continuous pipeline has none. You can set the schedule when you create the pipeline and change it on the pipeline’s page; a running pipeline restarts on its own to use the new one.
When a source table’s shape changes
Section titled “When a source table’s shape changes”A column added, dropped or renamed, a type changed, a truncate, or a table replaced at your source
is handled by the pipeline’s change policy: see
When a source table’s shape changes, which says what each answer does on
Snowflake. By default, an added column is added to your Snowflake table and its existing values
are copied; a dropped column is kept, with its values; a NUMBER that needs more digits becomes
NUMBER(38, s) at the same scale; a column whose new type no wider type of it can hold is
converted, every value checked, and becomes the table’s last column; and a truncate marks every row
the table held then as deleted (a table with no primary key kept live has them removed instead, whatever the rule).
What stops a pipeline today
Section titled “What stops a pipeline today”A stopped pipeline says why in its history. These stop a Snowflake pipeline and wait for you; it does not restart on its own, because every restart would resume your warehouse to meet the same refusal:
- A change to a table’s shape that the pipeline is set to ask you about, or one Snowflake
cannot make on its own. That table waits for you (it is parked); the pipeline’s other tables
carry on, unless it is set to stop every table when one stops: then every other table that is
syncing stops with it, also when the change is found as a Snowflake session opens, and resumes
with it (a table whose place is let go meanwhile waits for you to resume it, which copies it
again; see When one table stops). Its drawer on the Schema tab names
each change and the answers it takes; answer each, then Resume with these answers. If the
table has a place in your source’s log to carry on from, that place is kept while it waits, so
your source keeps the log it needs for that table until you act (if the source runs short of
room for it, the place is let go, and answering copies the table again). A table with no place
yet, and a table waiting because its key changed (
key_changed) or because it was dropped and created again (table_replaced), is copied again once you answer: a row your source no longer has is marked deleted (_REYANSH_DELETED_AT) in soft delete and removed in live, with or without a primary key, and that copy is billed like any other work. A table waiting for you raises an alert. Every other change to a table’s shape, and aTRUNCATE, is followed on its own: see When a source table’s shape changes. - A key value Snowflake cannot hold, or a number too wide for its column (above).
- A schema or table the role cannot write: a grant is missing, or the table has a shape we did not create. The stop names the schema or table, and the grant that is missing.
- The key or the role is refused by Snowflake. A refusal that comes from the clock of the machine running the pipeline is not this: Snowflake refuses a sign-in dated more than about 30 seconds ahead of its own clock, and from 29 seconds ahead the pipeline says the machine’s clock is ahead, by how much, and restarts once it is put right.
- A read of your source’s catalogue that PostgreSQL refuses: a privilege the source user
lacks, for example. The stop quotes PostgreSQL’s code and message and says what to grant.
A source user with
REPLICATIONandSELECTon the table is enough: it does not need to be a superuser or a member ofpg_read_all_settings.
Snowflake being slow, throttled or unreachable does not stop the pipeline for you: it retries on its own and its history says so.
Leaving
Section titled “Leaving”- Delete a pipeline: see Deleting a pipeline. Its tables stay in your Snowflake database; they are your data.
- To stop us reaching Snowflake at all:
ALTER USER "<user>" UNSET RSA_PUBLIC_KEY;, andUNSET RSA_PUBLIC_KEY_2too if a rotation is under way, or drop the user and the role. - What stays in your Snowflake account until you drop it: your tables; the working schema
beside each schema a pipeline wrote (
_REYANSH_<SCHEMA>, see Where a pipeline writes), which is safe to drop once no pipeline writes that schema; a test schema,_REYANSH_check_<time>_<random>, when a connection test could not drop its own (see A test schema left behind); and the user, the role and the warehouse the setup script created. - What stays in your account with us: the destination and its stored key. There is no way to remove a stored destination or credential yet. Unsetting the public key in Snowflake, as above, makes the stored key useless.