Storage and S3

Plan the persistent volumes and the object store. Make these decisions before the first install. Some cannot change later.

Storage class

Every volume claim uses the default StorageClass of the cluster (on k3s, local-path). To use another provisioner, set storageClass at the top level of local-env.yaml:

storageClass: longhorn

⚠️ Set it before the first install. A claim keeps its class. A change later makes the upgrade fail. It does not move the data.

The tile server and Photon need a class name. If the cluster has neither local-path nor standard, set storageClass even when your provisioner is the cluster default. Each component also takes its own storageClass. See the Storage section of local-env.template.yaml.

Size the volumes

On local-path, the size of a claim is only a label. On Longhorn or a cloud disk, it is the real capacity. Size these claims for your data:

Setting	Default	Holds
seaweedfs.volume.storageSize	50Gi per volume server, 3 servers	Every file that the platform stores
clickhouse.storage.size	150G with bigData, else 20G	Entities and relationships
opensearch.storage.size	30Gi	The search index
postgresql.storageSize	8Gi	Temporal and platform metadata
monitoring.prometheus.storage.size	30Gi	14 days of metrics

The optional map data needs much more. With tileserver.enabled or photon.enabled, the node holds about 280 GiB for the tiles and 240 GiB for the geocoding index while they copy. Size the disk for the images too. The install waits up to 240 minutes for these downloads. Add --no-wait-downloads to finish without the wait.

SeaweedFS keeps one copy of each file by default. On a cluster with at least two nodes, set seaweedfs.filer.defaultReplicaPlacement: "001" for two copies at twice the space. It applies to files that you write after the change.

Grow a volume

Your StorageClass must allow expansion (Longhorn does).

• ClickHouse and PostgreSQL: raise clickhouse.storage.size or postgresql.storageSize, then run ./bin/update-existing.sh. Do not edit the PVC.
• Redis, MySQL, OpenSearch, SeaweedFS, Loki: edit spec.resources.requests.storage of the PVC. Leave the size setting unchanged, or the next upgrade fails.
• Prometheus: edit the PVC. Leave monitoring.prometheus.storage.size unchanged. The extra space is room, not more history.

Longhorn on a single node

The stock longhorn class keeps 3 replicas, so each volume is Degraded on one node. Make a one-replica class and name it in storageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-single
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
parameters:
  numberOfReplicas: "1"
  dataLocality: "best-effort"
  fsType: ext4

Remove an install

./bin/destroy.sh --context <context-name> deletes the namespace and its claims.

⚠️ Data loss — With a Delete reclaim policy, destroy.sh removes the data. Take a backup first. See Backup & restore.

With Retain, the volumes stay. On a single node, delete the released PVs and their Longhorn volumes before you install again, or the new volumes may not fit.

External S3 (bring your own)

By default, SeaweedFS runs in the cluster as the S3 store. To use your own S3 (AWS S3, MinIO, Ceph RGW, or Google Cloud Storage through its S3 interop API), set:

s3:
  backend: external

externalS3:
  endpoint: "https://s3.eu-central-1.amazonaws.com"
  region: "eu-central-1"          # the real bucket region on AWS
  accessKeyId: "AKIAEXAMPLE"
  secretAccessKey: "REPLACE_ME"
  sharedBucketName: "acme-octostar-data"

langfuse:
  s3:
    bucket: "acme-langfuse"
postgresql:
  cnpg:
    backupBucket: "acme-cnpg-backups"

SeaweedFS is then not installed. The template has the full list of keys.

• Create the PostgreSQL backup bucket yourself, before the install. The platform creates the other buckets on first use. The access key needs CreateBucket, read and write.
• S3 bucket names are global. The default names are probably taken, so use your own.
• For a privately signed endpoint, set externalS3.caCert (inline PEM) or externalS3.caCertSecret (a Secret with the bundle under ca.crt). The CA needs a Subject Key Identifier, and the store's certificate needs an Authority Key Identifier.
• backup.remoteS3 (where backups go) takes a region, as in Backup & restore. The limit is the backup job's own view of your external store: it has no region and no CA. If your store is AWS outside us-east-1, or privately signed, the backup job cannot copy the buckets, ClickHouse and OpenSearch. PostgreSQL backups still work. Ask Octostar before you rely on backups with such a store.

⚠️ Moving a running install to external S3 uninstalls SeaweedFS and copies nothing. update-existing.sh asks you to confirm. Copy every bucket first. Also copy the audit bucket if audit.enabled is on, and the Loki bucket if you want old logs:

mc mirror --overwrite seaweedfs/octostar-data external/<externalS3.sharedBucketName>
mc mirror --overwrite seaweedfs/langfuse      external/<langfuse.s3.bucket>
mc mirror --overwrite seaweedfs/cnpg-backups  external/<postgresql.cnpg.backupBucket>
mc diff seaweedfs/octostar-data external/<externalS3.sharedBucketName>   # no output

If it fails

Symptom	Likely cause
SignatureDoesNotMatch on upload	Wrong externalS3.region for an AWS bucket
CERTIFICATE_VERIFY_FAILED on workspace import	The CA is not set
Uploads work, workspace import fails	The CA lacks a Subject Key Identifier, or the store certificate lacks an Authority Key Identifier
NoSuchBucket, or the install waits on PostgreSQL	A bucket does not exist
walArchivingFailing on PostgreSQL	postgresql.cnpg.backupBucket does not exist
The install stops and names externalS3.*	The endpoint or credentials are not set, or verify: false without a CA