Terraform: sources, destinations and pipelines as code
The Terraform provider manages the account’s credentials, sources, destinations and pipelines from configuration, applied by Terraform or OpenTofu like the rest of your infrastructure. It calls the public API with an API key, exactly as your own script would, so everything here is also true of the API.
Not yet in the public registry, so not yet installable. The provider is built and tested; publishing it needs a registry namespace and a signing key, which are being set up. This page describes it as it will ship, and the resource names (
reyansh_pipelineand the rest) are the ones the registry release will keep.
Requirements
Section titled “Requirements”- Terraform 1.11 or later, or OpenTofu 1.11 or later. Every secret is a write-only argument (below), which older versions cannot send; they refuse the configuration rather than store the secret.
- An API key of the account. A full key applies; a read-only key can
plan(it reads) and is refused atapply. Creating a destination is an owner’s power: use a full key an owner made.
Configure the provider
Section titled “Configure the provider”terraform { required_providers { reyansh = { source = "get-reyansh-ai/reyansh" } }}
provider "reyansh" { # Both may come from the environment instead: REYANSH_ENDPOINT and REYANSH_API_KEY. endpoint = "https://api.example.com" # your API's origin, no path}There is no default endpoint: name the API you mean, ours or your own control plane’s. Keep the key
out of .tf files: set REYANSH_API_KEY in your CI’s secret store.
Secrets are write-only, and never in state
Section titled “Secrets are write-only, and never in state”A secret (a connection string, a private key, a password) goes in an argument ending _wo. Terraform
sends it to the API and keeps it in no plan and no state file, so whoever can read your state
cannot read your database password. The API keeps it sealed and shows a masked hint.
Because Terraform does not store the secret, it cannot see you change it. To send a new secret, change it and raise its version number in the same apply (setting a version for the first time on an object Terraform created sends the secret too):
resource "reyansh_credential" "warehouse" { ref = "warehouse-postgres" value_wo = var.warehouse_url # a connection URL secret_wo_version = 2 # was 1: this apply rotates the credential}A rotation keeps the credential’s ref, so nothing that names it changes. A connection string a
running pipeline uses is rotated only to the same database (the API checks the server’s identity,
see the PostgreSQL destination guide). When the API cannot tell, it asks
for a confirmation that Terraform does not give: the apply stops with identity_unproven, and the
rotation is finished in the app.
The resources
Section titled “The resources”reyansh_credential
Section titled “reyansh_credential”A stored secret, named by your ref: a connection URL (value_wo), a key pair’s private key
(private_key_wo, with passphrase_wo when it is encrypted), or a Snowflake key pair the API
generates for you (generate = "rsa_2048"): then the private key never leaves us, and
public_key is what you register on the Snowflake user. A generated key pair is rotated in the app
(generate, test, switch), not by Terraform.
reyansh_source
Section titled “reyansh_source”A PostgreSQL database the account reads from: host, port, database, user, password_wo (and
password_wo_version), tls_mode and tls_trust (aws_rds where the account offers it). An edit to the connection is tested by the API
before it is saved, and every pipeline reading the source restarts onto it. A change of host, port
or TLS sends the password again, as the API requires. name and database are fixed after
creation.
reyansh_destination
Section titled “reyansh_destination”Where the data goes: kind = "postgres" on a connection-URL credential, or kind = "snowflake" on a
key-pair credential with settings (account, user, role, warehouse, database,
time_travel_days, wide_numeric_as). A Snowflake destination’s warehouse, role and user change in
place, after the API checks the new settings may keep every table its pipelines write; a new user
takes the credential_ref of the key registered on that user, in the same apply. account,
database, time_travel_days and wide_numeric_as are fixed after creation.
reyansh_pipeline
Section titled “reyansh_pipeline”A source’s tables kept in a destination:
resource "reyansh_pipeline" "orders" { id = "orders-to-warehouse" source_id = reyansh_source.shop.id destination_id = reyansh_destination.warehouse.id tables = [{ schema = "public", name = "orders" }, { schema = "public", name = "customers" }] historical_load = "requested" # copy the rows that already exist; or "declined" schedule = { kind = "cron", cron = "0 3 * * *", timezone = "Europe/London" } running = true}- Created stopped.
running = truestarts it; starting does not wait for the first copy, which can take hours.running = falsestops it after its batch. tables: the tables Terraform manages. Adding or removing one is a request the pipeline applies when it next starts; a running pipeline restarts onto it within a minute. A removed table’s rows stay in your destination. A table the pipeline adds on its own (its schema-change policy adds a new source table by default) is not intablesand is left alone: Terraform neither lists nor removes it. One you list and remove outside Terraform is added back by the next apply.- Not attributes yet: a pipeline’s
tablesare{ schema, name }only, so a table’s sync mode and a view’skey_columns(a query-based pipeline) are not set here, and a Snowflake destination has notrim_whitespace. Set them in the app or through the API; the provider never sends them, so an apply leaves them as they are. schedule:interval(15mto24h),cron(with atimezone, UTC when absent) ormanual. Absent on a new pipeline: every six hours.historical_load: leave it out and the first run stops and asks, as the app does. The provider never answers it for you.on_discarded_row,on_rejected_valueandrecord_in_destination: as rows your destination discards and values your destination cannot take describe.source_id,destination_id,update_method,destination_schema_namingandhistorical_loadare fixed after creation.
reyansh_egress_addresses (data source)
Section titled “reyansh_egress_addresses (data source)”The addresses our workers connect from, to allow in your database’s firewall, in the same configuration as the source that needs them:
data "reyansh_egress_addresses" "workers" {}
resource "aws_security_group_rule" "reyansh" { type = "ingress" from_port = 5432 to_port = 5432 protocol = "tcp" cidr_blocks = data.reyansh_egress_addresses.workers.ipv4_cidrs security_group_id = aws_security_group.db.id}cidrs holds every address, ipv4_cidrs and ipv6_cidrs each family: the addresses in use now.
A change of address is announced at least 30 days ahead (GET /v1/network’s upcoming);
a firewall built from this data source changes when you apply after the change, so apply again
within that window. A self-hosted deployment publishes none: published is false and the lists are
empty. A read-only key suffices.
A field fixed after creation, and why the plan refuses it
Section titled “A field fixed after creation, and why the plan refuses it”Changing a fixed field (a pipeline’s source_id, a source’s database) under the same id is
refused at plan, naming the field, and nothing is changed:
Error: Fixed after creation`source_id` cannot be changed on an existing pipeline: the API keeps it fixed ... To change it,give the pipeline a new `id`.Other providers replace the object instead: delete, then create. Here that would be a trap: a
pipeline’s id is never reused after a delete, so the create would be refused and you would be left
with the old pipeline deleted and no new one. Give the object a new id and Terraform replaces it
properly: the new pipeline is created under the new id, and the old one is deleted (its destination
tables are kept).
Destroy: what is deleted, and what is kept
Section titled “Destroy: what is deleted, and what is kept”- A pipeline is deleted (final; its id is not reused). Its destination tables, history and everything on the source are kept. Terraform prints, as a warning, the statements that remove its replication slot and publication from your source: run them yourself when nothing else uses them. We never change anything on a source, and a slot nothing reads holds write-ahead log until the source’s disk fills.
- A source, destination or credential is kept in the account. No API route deletes one yet, so
Terraform stops managing it and says so in a warning:
The source "shop" is kept in the account. Import it again to manage it. Order your destroys withdepends_on(or references, as above) so a pipeline goes before what it uses.
Import
Section titled “Import”Every resource imports by its id (a credential by its ref). An imported pipeline’s tables is every
table it has, until you list them:
terraform import reyansh_pipeline.orders orders-to-warehouseor with an import block. Two things an import cannot read back, and what happens to them:
- A secret’s version. After an import the
_wo_versionin your configuration is adopted, not sent: the first apply does not rotate anything. Raise it later to send a new secret. If the configuration had no version at the import, the first one you set is adopted too; raise it once more to send. - A pipeline’s
historical_load. The API does not return it; the value in your configuration is adopted.
Retries and rate limits
Section titled “Retries and rate limits”The provider retries what is safe to retry, and waits for the Retry-After the API sends (seconds or
a date). A read is retried after a 429 or a server error. A change carries an Idempotency-Key and
is retried only under that same key, so the API replays it rather than doing it twice: after a rate
limit, after a request that never reached the API, and while its first attempt is still running. A
server error the API answered for a change is not retried (the API keeps that answer for the key), so
the apply stops and says so; applying again sends a new request. At most five attempts. Keys are
limited to 600 requests a minute (1,200 across an account’s keys), which a large apply can reach;
the provider waits rather than fails.
An apply interrupted after the API created something (the CI job killed mid-apply) leaves an
object Terraform did not record. The next apply is refused, Already exists, naming the id and how
to adopt it: terraform import. A deleted pipeline’s id is never reused, so that one cannot be
imported: give the pipeline a new id.
A create never fails after the object exists. If the API stored a value differently from what was configured, or a pipeline’s start was refused, the apply warns and records the object as configured; the next plan shows the difference. A refused start is asked again by the next apply; a field fixed after creation that the API stored differently is refused at plan, and the message names the value to write. Other providers mark such an object for replacement, which for a pipeline would be a delete.
Not managed by Terraform, by rule
Section titled “Not managed by Terraform, by rule”People, invitations, API keys and money (the wallet, top-ups, payment methods) are not resources. An API key cannot change them at all (API keys), so a resource for them would always fail.