Skip to main content

GPU deployments

If your host has an NVIDIA GPU, Octostar's ML pipeline components and Streamlit apps can use it. Versions must be consistent across the stack — host driver, CUDA, and the app image are tied together.

Tested baseline​

  • OS: Ubuntu 22.04
  • NVIDIA driver: 560+ (tested with 560.35.03)
  • CUDA: 12.6
  • App base image: nvidia/cuda:12.6.0-base-ubuntu22.04

1. Provision the host​

Install the NVIDIA drivers and confirm the GPU is visible:

sudo ubuntu-drivers install
nvidia-smi

nvidia-smi should report the driver and CUDA version, e.g.:

NVIDIA-SMI 560.35.03    Driver Version: 560.35.03    CUDA Version: 12.6

2. Verify GPU access from a container​

Confirm the cluster can schedule a GPU workload:

kubectl run nvidia-smi --restart=Never --rm -i --tty \
  --image nvidia/cuda:12.6.0-base-ubuntu22.04 -- nvidia-smi

You should see the same nvidia-smi output from inside the pod. (This requires the NVIDIA device plugin / GPU operator on the cluster.)

3. Configure GPU values​

Use the GPU configuration template as your starting point instead of the CPU one:

cp local-env.template-gpu.yaml local-env.yaml

The pipeline component blocks (objectDetection, documentExtractor, imageAugmentation, audioTranscription) expose GPU-placement knobs — for example:

objectDetection:
  faceDetection:
    device: ["cuda:0"]
    gpuFallback: true
  generalEmbeddings:
    device: ["cuda:0"]
    gpuFallback: true
  # gpu:
  #   requests: 1
  #   limits: 1
  # nodeSelector:
  #   nvidia.com/gpu.present: "true"

Set device: ["cuda:0"] (or "cpu"), gpuFallback, and the gpu.requests / gpu.limits and nodeSelector blocks per component as appropriate for your hardware. See Configuration for the full set of pipeline knobs (worker queues, dynamic batching, per-task limits).

4. Run a GPU-powered app​

When creating a Streamlit app from the Octostar UI, edit its manifest.yaml to reference the GPU image:

image: octostar/streamlit-apps-gpu:latest

A minimal sanity-check app body (after the sample app's imports):

from torch import cuda

assert cuda.is_available()
assert cuda.device_count() > 0

print(cuda.get_device_name(cuda.current_device()))

st.header("This is a GPU powered instance")
st.subheader(cuda.get_device_name(cuda.current_device()))

If cuda.is_available() is True and the device name prints, the GPU path is working end to end.

⚠️ Keep versions aligned — Driver / CUDA / image-version mismatches are the usual cause of GPU pods failing to start or silently falling back to CPU. Keep the host driver, CUDA toolkit, and the nvidia/cuda base image of your app images on compatible versions.