Skip to content

Terraform: sources, destinations and pipelines as code

The Terraform provider manages the account’s credentials, sources, destinations and pipelines from configuration, applied by Terraform or OpenTofu like the rest of your infrastructure. It calls the public API with an API key, exactly as your own script would, so everything here is also true of the API.

Not yet in the public registry, so not yet installable. The provider is built and tested; publishing it needs a registry namespace and a signing key, which are being set up. This page describes it as it will ship, and the resource names (reyansh_pipeline and the rest) are the ones the registry release will keep.

  • Terraform 1.11 or later, or OpenTofu 1.11 or later. Every secret is a write-only argument (below), which older versions cannot send; they refuse the configuration rather than store the secret.
  • An API key of the account. A full key applies; a read-only key can plan (it reads) and is refused at apply. Creating a destination is an owner’s power: use a full key an owner made.
terraform {
required_providers {
reyansh = { source = "get-reyansh-ai/reyansh" }
}
}
provider "reyansh" {
# Both may come from the environment instead: REYANSH_ENDPOINT and REYANSH_API_KEY.
endpoint = "https://api.example.com" # your API's origin, no path
}

There is no default endpoint: name the API you mean, ours or your own control plane’s. Keep the key out of .tf files: set REYANSH_API_KEY in your CI’s secret store.

Secrets are write-only, and never in state

Section titled “Secrets are write-only, and never in state”

A secret (a connection string, a private key, a password) goes in an argument ending _wo. Terraform sends it to the API and keeps it in no plan and no state file, so whoever can read your state cannot read your database password. The API keeps it sealed and shows a masked hint.

Because Terraform does not store the secret, it cannot see you change it. To send a new secret, change it and raise its version number in the same apply (setting a version for the first time on an object Terraform created sends the secret too):

resource "reyansh_credential" "warehouse" {
ref = "warehouse-postgres"
value_wo = var.warehouse_url # a connection URL
secret_wo_version = 2 # was 1: this apply rotates the credential
}

A rotation keeps the credential’s ref, so nothing that names it changes. A connection string a running pipeline uses is rotated only to the same database (the API checks the server’s identity, see the PostgreSQL destination guide). When the API cannot tell, it asks for a confirmation that Terraform does not give: the apply stops with identity_unproven, and the rotation is finished in the app.

A stored secret, named by your ref: a connection URL (value_wo), a key pair’s private key (private_key_wo, with passphrase_wo when it is encrypted), or a Snowflake key pair the API generates for you (generate = "rsa_2048"): then the private key never leaves us, and public_key is what you register on the Snowflake user. A generated key pair is rotated in the app (generate, test, switch), not by Terraform.

A PostgreSQL database the account reads from: host, port, database, user, password_wo (and password_wo_version), tls_mode and tls_trust (aws_rds where the account offers it). An edit to the connection is tested by the API before it is saved, and every pipeline reading the source restarts onto it. A change of host, port or TLS sends the password again, as the API requires. name and database are fixed after creation.

Where the data goes: kind = "postgres" on a connection-URL credential, or kind = "snowflake" on a key-pair credential with settings (account, user, role, warehouse, database, time_travel_days, wide_numeric_as). A Snowflake destination’s warehouse, role and user change in place, after the API checks the new settings may keep every table its pipelines write; a new user takes the credential_ref of the key registered on that user, in the same apply. account, database, time_travel_days and wide_numeric_as are fixed after creation.

A source’s tables kept in a destination:

resource "reyansh_pipeline" "orders" {
id = "orders-to-warehouse"
source_id = reyansh_source.shop.id
destination_id = reyansh_destination.warehouse.id
tables = [{ schema = "public", name = "orders" }, { schema = "public", name = "customers" }]
historical_load = "requested" # copy the rows that already exist; or "declined"
schedule = { kind = "cron", cron = "0 3 * * *", timezone = "Europe/London" }
running = true
}
  • Created stopped. running = true starts it; starting does not wait for the first copy, which can take hours. running = false stops it after its batch.
  • tables: the tables Terraform manages. Adding or removing one is a request the pipeline applies when it next starts; a running pipeline restarts onto it within a minute. A removed table’s rows stay in your destination. A table the pipeline adds on its own (its schema-change policy adds a new source table by default) is not in tables and is left alone: Terraform neither lists nor removes it. One you list and remove outside Terraform is added back by the next apply.
  • Not attributes yet: a pipeline’s tables are { schema, name } only, so a table’s sync mode and a view’s key_columns (a query-based pipeline) are not set here, and a Snowflake destination has no trim_whitespace. Set them in the app or through the API; the provider never sends them, so an apply leaves them as they are.
  • schedule: interval (15m to 24h), cron (with a timezone, UTC when absent) or manual. Absent on a new pipeline: every six hours.
  • historical_load: leave it out and the first run stops and asks, as the app does. The provider never answers it for you.
  • on_discarded_row, on_rejected_value and record_in_destination: as rows your destination discards and values your destination cannot take describe.
  • source_id, destination_id, update_method, destination_schema_naming and historical_load are fixed after creation.

The addresses our workers connect from, to allow in your database’s firewall, in the same configuration as the source that needs them:

data "reyansh_egress_addresses" "workers" {}
resource "aws_security_group_rule" "reyansh" {
type = "ingress"
from_port = 5432
to_port = 5432
protocol = "tcp"
cidr_blocks = data.reyansh_egress_addresses.workers.ipv4_cidrs
security_group_id = aws_security_group.db.id
}

cidrs holds every address, ipv4_cidrs and ipv6_cidrs each family: the addresses in use now. A change of address is announced at least 30 days ahead (GET /v1/network’s upcoming); a firewall built from this data source changes when you apply after the change, so apply again within that window. A self-hosted deployment publishes none: published is false and the lists are empty. A read-only key suffices.

A field fixed after creation, and why the plan refuses it

Section titled “A field fixed after creation, and why the plan refuses it”

Changing a fixed field (a pipeline’s source_id, a source’s database) under the same id is refused at plan, naming the field, and nothing is changed:

Error: Fixed after creation
`source_id` cannot be changed on an existing pipeline: the API keeps it fixed ... To change it,
give the pipeline a new `id`.

Other providers replace the object instead: delete, then create. Here that would be a trap: a pipeline’s id is never reused after a delete, so the create would be refused and you would be left with the old pipeline deleted and no new one. Give the object a new id and Terraform replaces it properly: the new pipeline is created under the new id, and the old one is deleted (its destination tables are kept).

Destroy: what is deleted, and what is kept

Section titled “Destroy: what is deleted, and what is kept”
  • A pipeline is deleted (final; its id is not reused). Its destination tables, history and everything on the source are kept. Terraform prints, as a warning, the statements that remove its replication slot and publication from your source: run them yourself when nothing else uses them. We never change anything on a source, and a slot nothing reads holds write-ahead log until the source’s disk fills.
  • A source, destination or credential is kept in the account. No API route deletes one yet, so Terraform stops managing it and says so in a warning: The source "shop" is kept in the account. Import it again to manage it. Order your destroys with depends_on (or references, as above) so a pipeline goes before what it uses.

Every resource imports by its id (a credential by its ref). An imported pipeline’s tables is every table it has, until you list them:

Terminal window
terraform import reyansh_pipeline.orders orders-to-warehouse

or with an import block. Two things an import cannot read back, and what happens to them:

  • A secret’s version. After an import the _wo_version in your configuration is adopted, not sent: the first apply does not rotate anything. Raise it later to send a new secret. If the configuration had no version at the import, the first one you set is adopted too; raise it once more to send.
  • A pipeline’s historical_load. The API does not return it; the value in your configuration is adopted.

The provider retries what is safe to retry, and waits for the Retry-After the API sends (seconds or a date). A read is retried after a 429 or a server error. A change carries an Idempotency-Key and is retried only under that same key, so the API replays it rather than doing it twice: after a rate limit, after a request that never reached the API, and while its first attempt is still running. A server error the API answered for a change is not retried (the API keeps that answer for the key), so the apply stops and says so; applying again sends a new request. At most five attempts. Keys are limited to 600 requests a minute (1,200 across an account’s keys), which a large apply can reach; the provider waits rather than fails.

An apply interrupted after the API created something (the CI job killed mid-apply) leaves an object Terraform did not record. The next apply is refused, Already exists, naming the id and how to adopt it: terraform import. A deleted pipeline’s id is never reused, so that one cannot be imported: give the pipeline a new id.

A create never fails after the object exists. If the API stored a value differently from what was configured, or a pipeline’s start was refused, the apply warns and records the object as configured; the next plan shows the difference. A refused start is asked again by the next apply; a field fixed after creation that the API stored differently is refused at plan, and the message names the value to write. Other providers mark such an object for replacement, which for a pipeline would be a delete.

People, invitations, API keys and money (the wallet, top-ups, payment methods) are not resources. An API key cannot change them at all (API keys), so a resource for them would always fail.