Backup & restore

How to enable scheduled backups of a running Octostar deployment to an S3-compatible bucket, and how to restore a deployment from one.

Enabling backups

This section explains how to enable scheduled backups for a deployed Octostar environment and ship them to an S3-compatible bucket. The worked example targets a real AWS S3 bucket in eu-west-1 (Ireland), but the same steps apply to any S3-compatible store (Hetzner, Backblaze, Cloudflare R2, …).

📝 Backups are disabled by default — They turn on automatically as soon as backup.remoteS3.endpoint is set to a non-empty value in local-env.yaml. The same gate controls the backup CronJob, the credentials Secret, and PostgreSQL/CNPG WAL archiving.

What gets backed up

Component	Mechanism	Destination
MySQL	daily dump via the `octostar-backup` CronJob	local in-cluster S3 → remote S3
ClickHouse	daily dump via the `octostar-backup` CronJob	local in-cluster S3 → remote S3
OpenSearch	daily snapshot via the `octostar-backup` CronJob	local in-cluster S3 → remote S3
PostgreSQL (CNPG)	Barman WAL archiving + scheduled base backup (2 AM, 7-day retention)	remote S3 (`cnpg-backups` prefix)

The CronJob first dumps MySQL / ClickHouse / OpenSearch into the in-cluster object store (SeaweedFS), then copies all backup buckets up to the remote S3 endpoint you configure below.

Step 1 — S3 configuration (AWS)

1a. Create the bucket

Create a dedicated bucket in the region you intend to use, e.g. octostar-backups in eu-west-1.

1b. Create a dedicated IAM user + least-privilege policy

Create a dedicated IAM user (e.g. octostar-backups-rw) used only for this backup job, and attach a policy scoped to the single bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListBucket",
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>"
    },
    {
      "Sid": "ObjectRW",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>/*"
    }
  ]
}

Replace <BUCKET_NAME> with your bucket (e.g. octostar-backups).

1c. Generate an access key

When creating the access key, choose the "Application running outside AWS" use case — the backup CronJob runs in your own Kubernetes cluster outside AWS. (This selection is informational only; AWS will suggest IAM Roles Anywhere as an alternative but still issues a standard access key id + secret.)

Record the Access key ID and Secret access key securely. Do not commit them anywhere — they go only into local-env.yaml, which is gitignored.

Step 2 — Configure `local-env.yaml`

Add the following block. Setting endpoint to a non-empty value is what enables backups.

backup:
  schedule: "0 3 * * *"          # 3 AM daily (optional; this is the default)
  remoteS3:
    endpoint: "https://s3.eu-west-1.amazonaws.com"
    region: "eu-west-1"          # MUST match the bucket region for real AWS S3
    bucket: "octostar-backups"
    accessKeyId: "<FILL IN>"
    secretAccessKey: "<FILL IN>"

Endpoint reference for other providers:

Provider	Endpoint example
AWS	`https://s3.<region>.amazonaws.com`
Hetzner	`https://fsn1.your-objectstorage.com`
Backblaze	`https://s3.us-west-002.backblazeb2.com`

⚠️ Region matters on AWS — region is the SigV4 signing region and must equal the bucket's region (e.g. eu-west-1), or AWS rejects uploads with "the region 'us-east-1' is wrong; expecting '<region>'". The default us-east-1 is fine only for region-agnostic stores.

Step 3 — Test the credentials before applying

Verify the credentials and permissions out-of-band (e.g. from a laptop with the AWS CLI) before rolling them into the cluster. Export the key/secret as environment variables — do not paste them into files.

export AWS_ACCESS_KEY_ID='<key>'
export AWS_SECRET_ACCESS_KEY='<secret>'
export AWS_DEFAULT_REGION='eu-west-1'
BUCKET=octostar-backups
KEY="octostar-cred-test/probe-$$.txt"

# Identity
aws sts get-caller-identity

# List (should succeed; bucket may be empty)
aws s3api list-objects-v2 --bucket "$BUCKET" --max-items 5

# Put / Get / Delete round-trip
echo "octostar backup credential test" > /tmp/probe.txt
aws s3api put-object    --bucket "$BUCKET" --key "$KEY" --body /tmp/probe.txt
aws s3api get-object    --bucket "$BUCKET" --key "$KEY" /tmp/probe-get.txt
aws s3api delete-object --bucket "$BUCKET" --key "$KEY"
rm -f /tmp/probe.txt /tmp/probe-get.txt

All five operations (identity, list, put, get, delete) must succeed. The user ARN returned by get-caller-identity should be the dedicated backup user.

📝 Note — If the bucket has default SSE-S3 encryption enabled, put-object will report ServerSideEncryption: AES256 — this is expected and needs no configuration; AWS applies the bucket default transparently.

Step 4 — Apply to the running environment

Run update-existing.sh to configure the running environment:

cd /opt/octostar
./bin/update-existing.sh --context <KUBERNETES_CONTEXT>

Step 5 — Verify it works

# CronJob exists and is scheduled
kubectl get cronjob -n octostar-main

# Credentials Secret was created
kubectl get secret remote-s3-credentials -n octostar-main

# CNPG scheduled backup status
kubectl get scheduledbackup -n octostar-main

# Trigger an ad-hoc run instead of waiting for 3 AM
cd /opt/octostar
./bin/backup-now.sh --context <KUBERNETES_CONTEXT>

After a successful run, objects should appear in the remote bucket.

Configuration reference

Full set of backup keys (defaults shown):

backup:
  schedule: "0 3 * * *"            # CronJob schedule
  mysqlHost: "mysql"               # MySQL service name
  snapshotMaxBytesPerSec: "80mb"   # OpenSearch per-node snapshot rate cap
  restoreMaxBytesPerSec: "160mb"   # OpenSearch per-node restore rate cap
  remoteS3:
    endpoint: ""                   # non-empty => backups enabled
    region: "us-east-1"            # set to bucket region for AWS
    bucket: "os-octostar-backups"
    accessKeyId: ""
    secretAccessKey: ""

📝 Security — Credentials live only in local-env.yaml (gitignored) and the in-cluster remote-s3-credentials Secret. Never commit them to the repository or paste them into shared documents.

Restoring from backup

This section covers restoring an Octostar environment from a remote S3 backup using bin/restore-install.sh.

What this script does: it performs a complete, fresh install and then restores all application data from the remote S3 backup (SeaweedFS object data, MySQL, ClickHouse, OpenSearch, and PostgreSQL/CNPG via point-in-time recovery). It is a disaster-recovery (DR) tool — run it against an empty/fresh cluster, not on top of a healthy deployment you want to keep.

📝 No local-env.yaml required — Credentials are passed as command-line arguments, so no pre-existing local-env.yaml is required — the script fetches the backed-up local-env.yaml (and optionally versions.yaml) from the bucket's config-backups/ prefix.

Prerequisites

The remote S3 bucket must contain a valid backup (including config-backups/local-env.yaml).
kubectl, helm, helmfile, yq, and rclone must be installed — or pass --install-tools to install them.
A reachable Kubernetes context pointing at the target (fresh) cluster.

Basic restore command

./bin/restore-install.sh --context default \
  --s3-endpoint "https://s3.eu-west-1.amazonaws.com" \
  --s3-bucket "octostar-backups" \
  --s3-access-key "<ACCESS_KEY_ID>" \
  --s3-secret-key "<SECRET_ACCESS_KEY>"

📝 Region handling — For an AWS endpoint of the form s3.<region>.amazonaws.com, the SigV4 signing region is parsed automatically from the URL — you usually don't need --s3-region. For non-AWS / region-agnostic endpoints it defaults to us-east-1. Pass --s3-region explicitly for any AWS bucket outside us-east-1 whose region can't be parsed from the endpoint, otherwise both the config fetch and the in-cluster restore Job fail with AuthorizationHeaderMalformed.

Required arguments

Flag	Description
`--context NAME`	Kubernetes context to restore into
`--s3-endpoint URL`	Remote S3-compatible endpoint (e.g. `https://s3.eu-west-1.amazonaws.com`)
`--s3-bucket NAME`	Remote S3 bucket holding the backup
`--s3-access-key KEY`	Remote S3 access key ID
`--s3-secret-key KEY`	Remote S3 secret access key

Optional arguments

Flag	Description
`--s3-region REGION`	SigV4 signing region. Defaults to the region parsed from an AWS endpoint, else `us-east-1`. Required for AWS buckets outside `us-east-1` when it can't be derived from the endpoint.
`--keep-local-env`	Do not overwrite `local-env.yaml` with the backup's copy — keep the target cluster's existing config (air-gap registries, certs, ingress) and restore data only. Requires an existing `local-env.yaml`. Use for cross-environment restores. (S3 creds/region are still patched in so the restore Job can reach the bucket.)
`--domain FQDN`	Restore onto a different domain than the backup was taken on (cross-domain DR, e.g. `home.staging-aws.octostar.com`). The backed-up `local-env.yaml` is fetched, then `octostar.domain` is overridden before any helm rendering, so ingress/TLS/`APP_HOST` come up on the new domain. Omit to keep the backup's domain.
`--offline`	Air-gapped restore: routes the install phase through `offline-installer.sh` (serves charts from a local bundle) instead of `install.sh`. Requires `offline-installer.sh` + helpers in `bin/` and the chart `.tgz` bundle at `OCTOSTAR_CHARTS_OFFLINE` (default `~/octostar-singlenode/charts-offline`).
`--use-backed-up-versions`	Restore using the component versions that were live at backup time (fetches `config-backups/versions.yaml` from the backup) instead of this checkout's `versions.yaml`. Your current file is saved as `versions.yaml.bak`. Off by default; fails fast if the backup has no version snapshot. With `--offline`, the offline bundle must contain those chart versions.
`--install-tools`	Install required tools (`kubectl`, `helm`, `helmfile`, `yq`, `rclone`) before restoring
`--install-utils`	Install utilities (`k9s`, `stern`)
`--log-level LEVEL`	`info` (default), `debug`, `warn`, `error`. Also forwarded to `install.sh`.
`-h`, `--help`	Print usage and exit

Any other unrecognised options are forwarded as-is to install.sh (run ./bin/install.sh --help for that list).

⚠️ No dry-run — --dry-run is not supported (and the script aborts if DRY_RUN is set in the environment). The restore phases always run for real against the cluster, so there is no safe dry-run mode.

Common scenarios

Same-cluster / same-domain DR — full restore onto a fresh cluster, keeping the backup's domain and config:

./bin/restore-install.sh --context default \
  --s3-endpoint "https://s3.eu-west-1.amazonaws.com" \
  --s3-bucket "octostar-backups" \
  --s3-access-key "<ACCESS_KEY_ID>" --s3-secret-key "<SECRET_ACCESS_KEY>"

Cross-domain restore — bring the data up on a new domain:

./bin/restore-install.sh --context default \
  --s3-endpoint "https://s3.eu-west-1.amazonaws.com" \
  --s3-bucket "octostar-backups" \
  --s3-access-key "<ACCESS_KEY_ID>" --s3-secret-key "<SECRET_ACCESS_KEY>" \
  --domain "home.staging-aws.octostar.com"

Cross-environment restore (data only) — keep the target cluster's existing local-env.yaml:

./bin/restore-install.sh --context default \
  --s3-endpoint "https://s3.eu-west-1.amazonaws.com" \
  --s3-bucket "octostar-backups" \
  --s3-access-key "<ACCESS_KEY_ID>" --s3-secret-key "<SECRET_ACCESS_KEY>" \
  --keep-local-env

Bootstrapping a clean machine — install all tooling first:

./bin/restore-install.sh --context default \
  --s3-endpoint "https://s3.eu-west-1.amazonaws.com" \
  --s3-bucket "octostar-backups" \
  --s3-access-key "<ACCESS_KEY_ID>" --s3-secret-key "<SECRET_ACCESS_KEY>" \
  --install-tools --install-utils

What happens during the restore (5 phases)

The script is non-interactive and streams progress for the long-running phases:

Step 0 — Fetch config: downloads config-backups/local-env.yaml from the bucket (unless --keep-local-env), patches in the S3 creds/region, and sets the transient DR flags.
Step 1 — Fresh install: runs install.sh --restore-mode (Temporal/ Langfuse deferred until Postgres is back).
Step 2 — Data restore: deploys the restore Job (Helm hook, up to a 6h timeout) to rehydrate object storage, MySQL, ClickHouse, and OpenSearch; then quiesces and restarts OpenSearch with normal security.
Step 3 — PostgreSQL / CNPG recovery: redeploys CNPG in recovery mode and waits (up to 6h) for Cluster in healthy state, then re-enables normal mode and installs the deferred Temporal/Langfuse.
Step 4 — Initialize datasources: runs init-datasources in restore mode (license, JWT, timbr roles, ingestion pipelines) and syncs roles.
Step 5 — Finalize: stores the final local-env.yaml + versions.yaml in the local-env-backup Secret and re-enables the backup CronJob.

📝 Resumability & safety — Transient DR flags (octostar.restoreMode, opensearch.restoreMode, postgresql.cnpg.deferBackups, postgresql.cnpg.recovery) are always reset on exit — even on failure — via an EXIT trap, so a subsequent normal helmfile sync won't accidentally run in restore mode. The backup CronJob stays off for the whole restore and is only re-enabled at the very end, so a scheduled run can't overwrite the good remote DR copy mid-restore. If a phase times out, the script prints exact remediation commands; re-running it is safe (Step 3 deletes any half-recovered CNPG cluster so recovery re-bootstraps cleanly).

Verifying a successful restore

On success the script prints === Disaster recovery complete ===. If it instead prints === Disaster recovery completed WITH ERRORS === and exits non-zero, the data was restored but the backup CronJob could not be re-enabled — scheduled backups are NOT active. Re-enable them before relying on the cluster:

helmfile --kube-context <CONTEXT> -f helmfile-backup.yaml sync \
  --set restoreJob.enabled=false --set cronjob.enabled=true
kubectl --context <CONTEXT> -n octostar-main get cronjob octostar-backup

Quick post-restore checks:

kubectl --context <CONTEXT> -n octostar-main get pods
kubectl --context <CONTEXT> -n octostar-main get cluster.postgresql.cnpg.io cnpg-cluster
kubectl --context <CONTEXT> -n octostar-main get cronjob octostar-backup

📝 Security — Pass S3 credentials only on the command line of a trusted host (mind shell history) — never commit them. The script writes them into local-env.yaml (gitignored) and the in-cluster local-env-backup Secret only.

Enabling backups​

What gets backed up​

Step 1 — S3 configuration (AWS)​

1a. Create the bucket​

1b. Create a dedicated IAM user + least-privilege policy​

1c. Generate an access key​

Step 2 — Configure local-env.yaml​

Step 3 — Test the credentials before applying​

Step 4 — Apply to the running environment​

Step 5 — Verify it works​

Configuration reference​

Restoring from backup​

Prerequisites​

Basic restore command​

Required arguments​

Optional arguments​

Common scenarios​

What happens during the restore (5 phases)​

Verifying a successful restore​