How to use your NVIDIA GPU in Docker with the NVIDIA Container Toolkit?

Updated: 28 September 2026 · Applies to: Docker Engine 29 and NVIDIA Container Toolkit 1.20 on Ubuntu 24.04 / 26.04 LTS

Most AI projects ship as Docker images. To let a container use your NVIDIA GPU you install the NVIDIA Container Toolkit on the server and connect it to Docker. This guide assumes Ubuntu with the NVIDIA driver already working.

Before you start

nvidia-smi must print a table on the server itself (How to install the NVIDIA driver on Ubuntu 24.04 / 26.04 for a GPU server?). The container uses the host's driver; do not install a driver inside the container.
Docker Engine must be installed. Follow Docker's official instructions for Ubuntu at docs.docker.com/engine/install/ubuntu, then check it:
```
docker --version
```

Install the NVIDIA Container Toolkit

Install the tools the next step needs:

sudo apt-get update && sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg2

Add NVIDIA's repository and signing key:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Install the toolkit:

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

Tell Docker to use it and restart Docker:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test it

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

You should see the same GPU table as on the server itself. If you do, containers can use the GPU.

Choose which GPUs a container gets

All GPUs: --gpus all
Only GPU 0 and GPU 1 (numbers from nvidia-smi -L): --gpus '"device=0,1"'

The single quotes around the double quotes are required. Example:

sudo docker run --rm --gpus '"device=0"' ubuntu nvidia-smi

With Docker Compose

Add a GPU reservation to the service:

services:
  app:
    image: your-image
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Common problems

could not select device driver "" with capabilities: [[gpu]]: Docker does not know about the toolkit yet. Run the two commands of the last install step again and restart Docker.
nvidia-smi fails on the server: fix the driver first: How to fix the "NVIDIA-SMI has failed" error (cannot communicate with the NVIDIA driver)?.
The container starts but the program uses the CPU: you forgot --gpus in the docker run command, or the image was built without CUDA support.

Frequently asked questions

Do I need the CUDA Toolkit on the server for Docker?
No. Only the driver and the NVIDIA Container Toolkit. The CUDA libraries live inside the image.

Does this work with Open WebUI or other GPU images?
Yes. Add --gpus all to the docker run command of the image that needs the GPU.

Official documentation: NVIDIA Container Toolkit installation guide (tested with toolkit 1.20.1).

Need a GPU server, or a hand with the setup?

GPU dedicated servers: NVIDIA GPU servers for AI training and inference; our engineers can install the driver, CUDA, PyTorch or a private LLM and hand it over ready to use.
Private LLM installation: we install Ollama, the GPU driver and a chat interface on your own server.

Prefer a hand with the setup? Our engineers can do it for you: Hire an Expert, or use our on-demand server management.