Skip to main content

Operations & uninstallation

Use this page for day-2 checks, common fixes and removal of an install.

Common commands​

These run the release's own kubectl and helmfile: put them on your PATH first (see Prerequisites), or use your own of a matching version.

# Pod and service status (default namespace: octostar-main)
kubectl get pods -n octostar-main
kubectl get svc  -n octostar-main

# Logs and roll-out status of a workload
kubectl logs -n octostar-main deploy/<deployment>
kubectl rollout status deploy/<deployment> -n octostar-main

# Stream the namespace logs to the screen and to a timestamped file (needs stern)
kubectl config use-context <context-name>
./bin/install.sh --dump-logs --context <context-name>

--install-utils installs k9s and stern. --dump-logs reads the octostar-main namespace and runs until you press Ctrl+C. It ignores --context and reads the cluster of your current context, so switch to the right context first. Add --log-level debug to install.sh for more detail. update-existing.sh and destroy.sh do not accept this option.

To log in as the admin, see Keycloak and custom OIDC. Without Keycloak, use admin and timbrAdminPassword at https://home.<domain>.

Monitoring​

Grafana ships dashboards for ClickHouse, OpenSearch, PostgreSQL, SeaweedFS, Temporal and octostar-api.

⚠️ Alerts reach nobody by default — Set up delivery as described in Alert notifications. On a cluster that already runs Prometheus and Loki, see Installing on a shared cluster.

If something is wrong​

SymptomFix
Pod in ImagePullBackOffCheck dockerHub.token. On RKE2, check /etc/rancher/rke2/registries.yaml on every node. Air-gapped: see Air-gapped installation.
Pod in CrashLoopBackOffRead its logs: kubectl logs -n octostar-main <pod> --previous.
Pod stays PendingThe node has too little CPU, memory or storage. Run kubectl describe pod <pod>. See Prerequisites.
Pod killed (OOMKilled)The node has too little memory. This is the most common cause of a failed install.
UPGRADE FAILED … conflict with "kubectl-patch"helm 4 does not overwrite a field that someone changed by hand. Put the field back, or let the chart win for that release (below). Make changes in local-env.yaml to avoid this.

To let the chart win for one release, then run bin/update-existing.sh again, with the release's own helmfile and helm on your PATH (see Prerequisites):

helmfile --kube-context <ctx> -f <the helmfile named in the error> \
  -l name=<release> --args=--force-conflicts sync

Uninstalling​

./bin/destroy.sh --context <context-name>

destroy.sh uninstalls the releases of the install and deletes its namespaces. The Langfuse, backup and apps-upgrade releases go with the namespace. It does not delete a cert-manager that the cluster already had. On a minikube context it deletes the cluster. On docker-desktop it stops: reset that cluster from the Docker Desktop UI.

⚠️ Destructive — destroy.sh removes the Octostar deployment and its data from the targeted context. It asks you to type yes only when it runs in a terminal. Over ssh host ./bin/destroy.sh, in CI or in a pipe, it deletes at once with no prompt. Back up first (see Backup & restore).

⚠️ Persistent volumes can stay — With reclaimPolicy: Retain (for example the single-node Longhorn class), the volumes stay after the namespace is gone. On Longhorn they still count against the free space, so a new install on the same node may not fit. Delete the released volumes (PVs and Longhorn volumes) before you install again. The vLLM model weights also stay. Delete the vllm namespace to remove them.

Getting help​

Keep the --dump-logs file, and contact Octostar support.

Recovering from an interrupted mapping refresh​

A mapping refresh (POST or PATCH /api/v1/search_api/<ontology>/refresh-opensearch-mapping) answers 409 when a reindex was interrupted and left OpenSearch inconsistent: a _reindex task still running with no refresh behind it, aliases split across indices, or a data index without aliases. Recover the index before refreshing again, and do it right away: entity upserts go through the read alias (os_ontology_v1), but pipeline writes (processing status, materialized annotations, scripted updates) go through the write alias (os_ontology_v1_write), and OpenSearch refuses every one of them while that alias spans two indices ("no write index is defined for alias").

The examples use os_ontology_v1; substitute the ontology from the 409. To reach OpenSearch:

kubectl port-forward -n octostar-main svc/opensearch 9200 &
OS="https://localhost:9200"
AUTH="admin:<password from the opensearch-secret secret>"

0. Make sure no refresh is running​

A refresh that is still running looks exactly like an interrupted one: its _reindex task runs and its write alias spans both indices until its cleanup. Changing anything under it makes its cleanup and rollback delete both copies. Before going on, confirm:

  • the refresh's task in the Tasks list is no longer running, or, if it shows Running but nothing progresses, its Temporal workflow opensearch-reindex-<task id> is closed: temporal workflow describe --workflow-id opensearch-reindex-<task id>;
  • no reindex is underway on purpose (another operator, an upgrade).

1. Find the state​

curl -sk -u "$AUTH" "$OS/_cat/aliases/os_ontology_v1*?v"
curl -sk -u "$AUTH" "$OS/_cat/indices/os_ontology_v1*?v&h=index,docs.count,store.size,creation.date.string"
curl -sk -u "$AUTH" "$OS/_tasks?actions=*reindex&detailed=true"
What you seeWhat happenedIndex to keep
Read alias on A; write alias on A and BInterrupted before the cutover; B is a partial copyA
Read alias on B; write alias on A and BInterrupted after the cutover, before the cleanup; A is the complete previous copyB
An index named os_ontology_v1 and an os_ontology_v1_v<n> in the write alias; no read aliasThe first move to aliases was interrupted; the read alias is created only at its last stepos_ontology_v1 (the original)
No alias; one or more os_ontology_v1_v<n>A rollback after the cutover deleted the new index; the previous copy remains without aliasesThe one holding the documents (docs.count)

When in doubt, keep the index with the most documents and the latest data.

An extra os_ontology_v1_v<n> with no alias next to an index holding both aliases causes no 409: a rollback whose _reindex OpenSearch registered late recreated its destination as a partial copy with dynamic mappings. Delete it as in step 4.

2. Stop a stray reindex​

A _reindex task naming an os_ontology_v1 index has no worker left to cut over or clean up. Cancel it and wait until it stops, or it recreates an index deleted under it:

curl -sk -u "$AUTH" -X POST "$OS/_tasks/<task id>/_cancel?wait_for_completion=true"

3. Put both aliases on the index to keep​

curl -sk -u "$AUTH" -X POST "$OS/_aliases" -H 'Content-Type: application/json' -d '{
  "actions": [
    { "remove": { "index": "<other index>", "alias": "os_ontology_v1_write" } },
    { "remove": { "index": "<other index>", "alias": "os_ontology_v1" } },
    { "add": { "index": "<index to keep>", "alias": "os_ontology_v1" } },
    { "add": { "index": "<index to keep>", "alias": "os_ontology_v1_write" } }
  ]
}'

Drop the remove actions for aliases the other index doesn't have. For an interrupted first move to aliases, add nothing: remove the write alias from both indices instead, and the next refresh moves to aliases again. Pipeline writes resume as soon as the write alias is on a single index.

4. Delete the other index​

If you kept B after the cutover, A can hold entity edits B lacks: edits made during the copy went through the read alias, still on A, after _reindex took its snapshot. Copy them to B, or accept losing them, before deleting A.

Check once more that it is behind neither alias and is not the one you kept, then:

curl -sk -u "$AUTH" -X DELETE "$OS/<other index>"

5. Free the refresh slot​

While the interrupted run still holds its Redis slot, a refresh POST answers 202 with that run's task id, and that task stays Running forever; the PATCH answers 409 "reindex in progress". The slot expires on its own after the reindex timeout plus 10 minutes. Once step 0 confirmed nothing runs, free it at once in the platform's Redis:

redis-cli DEL octostar:opensearch_reindex:active

6. Refresh again​

Re-run the mapping refresh, or the live PATCH.