Backup & restore

Back up a running Octostar install to an S3-compatible bucket, and restore it onto a fresh cluster.

What is backed up

Backups are off by default. They start when backup.remoteS3.endpoint is set.

What	When (default)
MySQL (when your install uses it), ClickHouse (entities and relationships), OpenSearch, the SeaweedFS buckets, `local-env.yaml`, `versions.yaml` and the platform EC private key	Every day at 03:00
PostgreSQL (Temporal and platform metadata): base backup and continuous WAL, with point-in-time recovery	Base backup every day at 02:00, 7-day retention

Not backed up:

Loki logs. A restored install starts with empty logs. Send logs to another store if you must keep them.
Work in progress in the pipeline. After a restore, those entities have the state STALE. Process them again.
Anything outside these stores, for example host state or resources that you added by hand.

⚠️ The bucket holds your secrets in plaintext — It has local-env.yaml (database passwords, API keys, the S3 keys) and the EC private key. Whoever reads the bucket has full access to the platform. See Secure the bucket.

Enable backups

The example uses AWS S3 in eu-west-1. Other S3-compatible stores work the same way.

1. Create the bucket and a dedicated user

Create a bucket, for example octostar-backups. Create an IAM user used only for backups, and attach this policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListBucket",
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>"
    },
    {
      "Sid": "ObjectRW",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>/*"
    }
  ]
}

Create an access key for the user (use case: "Application running outside AWS"). Keep the key and the secret out of git.

2. Set `local-env.yaml`

backup:
  remoteS3:
    endpoint: "https://s3.eu-west-1.amazonaws.com"
    region: "eu-west-1"          # must equal the bucket region on AWS
    bucket: "octostar-backups"
    accessKeyId: "<FILL IN>"
    secretAccessKey: "<FILL IN>"

Other endpoints: Hetzner https://fsn1.your-objectstorage.com, Backblaze https://s3.us-west-002.backblazeb2.com. The default region us-east-1 is right only for stores that ignore the region.

3. Test the credentials

Run this from a laptop with the AWS CLI, before you apply them to the cluster:

export AWS_ACCESS_KEY_ID='<key>' AWS_SECRET_ACCESS_KEY='<secret>' AWS_DEFAULT_REGION='eu-west-1'
aws sts get-caller-identity
aws s3api list-objects-v2 --bucket octostar-backups --max-items 5
echo test > /tmp/probe.txt
aws s3api put-object    --bucket octostar-backups --key probe.txt --body /tmp/probe.txt
aws s3api delete-object --bucket octostar-backups --key probe.txt

All commands must succeed.

4. Apply and verify

The last line runs the release's own kubectl, so the line before it puts the release's tools on your PATH (see Prerequisites):

cd /opt/octostar
./bin/update-existing.sh --context <context-name>
./bin/backup-now.sh --context <context-name>     # run one backup now
export PATH="$PWD/bin/tools/$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/'):$PATH"
kubectl --context <context-name> -n octostar-main get cronjob,scheduledbackup

After the run, objects must appear in the bucket.

Restore

bin/restore-install.sh makes a complete fresh install, then restores the data from the bucket.

⚠️ Restore only onto a fresh, empty cluster. Do not run it on a deployment that you want to keep. There is no dry run.

You need a kube context for the new cluster. helm, helmfile, kubectl, yq and, from v2.0.10, rclone come with the release. With v2.0.9, install rclone yourself. With an older release, or on a platform the release has no tools for (an Intel Mac), install the missing tools yourself (see Prerequisites). You do not need a local-env.yaml. The script reads the backed-up one from the bucket.

./bin/restore-install.sh --context <context-name> \
  --s3-endpoint "https://s3.eu-west-1.amazonaws.com" \
  --s3-bucket "octostar-backups" \
  --s3-access-key "<ACCESS_KEY_ID>" \
  --s3-secret-key "<SECRET_ACCESS_KEY>"

Add these options when you need them:

Option	Use
`--s3-region REGION`	An AWS bucket outside `us-east-1` whose region is not in the endpoint URL. Without it, the restore fails with `AuthorizationHeaderMalformed`.
`--domain FQDN`	Restore onto a different domain. See Restore onto a different domain.
`--keep-local-env`	Keep the `local-env.yaml` of the target cluster (registries, certificates, ingress) and restore data only.
`--offline`, `--chart-repo URL`	Air-gapped restore. See Air-gapped restore.
`--use-backed-up-versions`	Use the component versions of the backup, not those of your checkout. The scripts stay at your checkout's revision. With `--offline`, the chart repository must hold the backed-up chart versions.
`--install-utils`	Install `k9s` and `stern` first, for debugging.

A restore often takes more than 30 minutes. Do not stop it. If a step times out, the script prints the commands to fix it, and you can run it again. The backup CronJob stays off until the end, so a scheduled run cannot overwrite your good backup.

Verify

The script prints === Disaster recovery complete ===. Then check, with the release's own kubectl, helmfile and helm on your PATH (see Prerequisites):

kubectl --context <context-name> -n octostar-main get pods
kubectl --context <context-name> -n octostar-main get cronjob octostar-backup

If it prints === Disaster recovery completed WITH ERRORS ===, the data is restored but scheduled backups are not active. Turn them on:

helmfile --kube-context <context-name> -f helmfile-backup.yaml.gotmpl sync \
  --set restoreJob.enabled=false --set cronjob.enabled=true

If it fails

AuthorizationHeaderMalformed: pass --s3-region.
The restore stopped inside the OpenSearch window. Check that the restore flags are off before you use the cluster, then sync OpenSearch with the release's own helmfile and helm on your PATH (see Prerequisites):
```
grep -n restoreMode local-env.yaml     # nothing, or restoreMode: false
helmfile --kube-context <ctx> -f helmfile.yaml.gotmpl --selector name=opensearch sync
```
The restore stops over the EC key. Two cases:
- The backup has no key file, and its local-env.yaml pins an empty key. The source ran on a generated key that the backup does not hold. You cannot restore the encrypted data without that key.
- You used --keep-local-env, the backup has no key file, and your local-env.yaml pins no key. Set octostar.ecPrivateKey to the key of the source install.
A backup with no key file and no pinned key restores with the old shared default key, and the script warns you. Rotate the key afterwards. See Upgrading.

Air-gapped restore

./bin/restore-install.sh --context <context-name> \
  --s3-endpoint "<url>" --s3-bucket "<bucket>" \
  --s3-access-key "<key>" --s3-secret-key "<secret>" \
  --offline --chart-repo "http://charts.example.internal:8879"

The chart repository must hold every chart, including the backup chart. See Air-gapped installation.
Your registry and mutator must serve the images. The backed-up local-env.yaml has the pinned PostgreSQL images of the old cluster. If the new registry differs, add --keep-local-env and set them yourself.

Restore onto a different domain

Add --domain <new-fqdn>. The ingress, the certificates and the public URLs use the new domain. The data does not change. Two things stay: absolute links that users saved in their content keep the old domain, and a different parent domain can need new redirect URIs on the OAuth client. Check that login works afterwards.

Point-in-time recovery

To recover PostgreSQL to a given time, restore with --keep-local-env and set the time in the local-env.yaml that you keep:

postgresql:
  cnpg:
    recoveryTargetTime: "2024-01-01T00:00:00Z"

⚠️ Without --keep-local-env the restore replaces your local-env.yaml with the backup's copy. The target time is lost, and PostgreSQL replays to the latest WAL. That can replay the very change that you want to undo. The script turns recovery mode on itself. Do not set recovery yourself.

After a PostgreSQL major upgrade

A PostgreSQL backup restores only onto the major version that wrote it. After a major upgrade, take a base backup at once. The new version has an empty archive:

./bin/backup-now.sh --context <context-name>

To restore an archive of an earlier major version, ask your Octostar contact for the steps.

Secure the bucket

Use dedicated, least-privilege credentials for the backup job. Do not use root keys. Limit the bucket policy to the principals that need it.
Turn on encryption at rest.
Turn on versioning, so a compromised cluster cannot erase good backups. Add a lifecycle rule that expires noncurrent versions after 30 days and removes expired delete markers. Without it, the bucket keeps every backup that you ever took.
--s3-secret-key shows in your shell history and in ps. Run the restore from a clean shell.
Rotate the keys on a schedule and after a suspected exposure. For a new key, change backup.remoteS3 in local-env.yaml and run update-existing.sh.

What is backed up​

Enable backups​

1. Create the bucket and a dedicated user​

2. Set local-env.yaml​

3. Test the credentials​

4. Apply and verify​

Restore​

Verify​

If it fails​

Air-gapped restore​

Restore onto a different domain​

Point-in-time recovery​

After a PostgreSQL major upgrade​

Secure the bucket​