Skip to main content

GPU deployments

Use this page to let Octostar use an NVIDIA GPU on a node: ML pipeline components, GPU apps, T3K CORE and a local LLM (vLLM).

⚠️ A pod without a GPU does not fail — If the NVIDIA runtime does not handle a container, the pod starts, runs on the CPU and reports no error. Always do the check from inside a pod (step 3).

What you need​

Octostar tested this setup. Other versions can work, but Octostar did not test them.

  • Ubuntu 22.04 and NVIDIA driver 560 (tested with 560.35.03)
  • CUDA 12.6 and the app base image nvidia/cuda:12.6.0-base-ubuntu22.04
  • NVIDIA container toolkit 1.18 or newer

1. Install the driver​

sudo ubuntu-drivers install
nvidia-smi

nvidia-smi must show the driver and CUDA version. This host check alone is not enough. Do step 3 too.

2. Make the NVIDIA runtime the default​

Octostar GPU workloads do not request nvidia.com/gpu and set no runtimeClassName. So the NVIDIA runtime must be containerd's default runtime. Install the toolkit from NVIDIA's apt repository (distribution packages can be older), then set the default:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
nvidia-container-runtime --version   # must report 1.18.0 or newer

sudo nvidia-ctk runtime configure --runtime=containerd --set-as-default
sudo systemctl restart containerd

On k3s, which has its own containerd, write the config template instead:

sudo nvidia-ctk runtime configure --runtime=containerd \
  --config=/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl --set-as-default
sudo systemctl restart k3s

Do not change these settings:

  • Leave the runtime mode at auto. Do not set mode = "legacy". With toolkit 1.17 or older, the next systemctl daemon-reload (for example an apt upgrade) removes the GPU from running GPU containers, and workers continue on the CPU with no error.
  • Do not set accept-nvidia-visible-devices-envvar-when-unprivileged = false in /etc/nvidia-container-runtime/config.toml. Octostar GPU apps then run on the CPU.

A node on toolkit 1.17 or older needs an upgrade and a restart of every GPU workload. See README-GPU.md in the octostar-singlenode repository.

3. Verify from inside a pod​

kubectl run nvidia-smi --restart=Never --rm -i --tty \
  --image nvidia/cuda:12.6.0-base-ubuntu22.04 -- nvidia-smi

You must see the same output as on the host.

4. Use the GPU template​

cp local-env.template-gpu.yaml local-env.yaml

The pipeline components in this template place work on devices such as cuda:0 and cuda:1.

⚠️ Check the device lists — Make sure every device that the template names exists on your node. With gpuFallback: true, a component continues on the CPU when it cannot use its device. See Configuration for the other settings.

To run a GPU Streamlit app, set image: octostar/streamlit-apps-gpu:latest in its manifest.yaml.

T3K CORE (image analysis)​

T3K CORE is off by default. Before you set t3k.enabled: true, you need:

  1. A license file (core.lic) from the vendor. Set exactly one of t3k.license.contentBase64, t3k.license.existingSecret or t3k.license.content.

  2. The two vendor images, loaded into the node's containerd by hand. They are in no registry. The tags must match versions.t3kCore.apiImageTag and kryptonImageTag:

    sudo k3s ctr -n k8s.io images import t3k_core_<ver>.tar
    sudo k3s ctr -n k8s.io images import krypton_<ver>.tar

  3. Kubernetes 1.29 or newer.

  4. A GPU and the default NVIDIA runtime (step 2).

  5. About 6 CPU and 24 Gi of free requests on one node. With less, the pod stays Pending and the install waits 30 minutes. Check with kubectl describe node | grep -A6 'Allocated resources'.

After the install, check the toolkit version by hand. The installer's own GPU check does not find the legacy-mode failure from step 2.

Local LLM (vLLM)​

Set vllm.enabled: true and a vllm.profile in local-env.yaml. The profiles are qwen36, qwen3635 and qwen3827. Every profile serves on two GPUs, so a node with one GPU cannot run vLLM. The vLLM pod downloads the model weights from Hugging Face at first start, so the cluster, not the install host, needs access to it. On a cluster with more than one node, set vllm.nodeSelector before the first install. Overrides are in README-GPU.md in the octostar-singlenode repository.

⚠️ Air-gapped installs — An air-gapped install can serve vLLM only from weights that are already on the volume, at the revision that the profile pins. A volume with another revision downloads again, and that fails offline. vllm.weights: image is refused.

If it fails​

  • A pod runs but is slow, with no error. It runs on the CPU. Do step 3 and check that the NVIDIA runtime is the default.
  • Failed to initialize NVML: Unknown Error. The toolkit is 1.17 or older. Upgrade it and restart the GPU workloads.
  • T3K CORE stays Pending. The node has too little free CPU or memory.
  • A GPU pod fails to start. The driver, CUDA and image versions do not match.