Serve LLM in production on DigitalOcean GPU Droplet with SGLang

Most "run an LLM in the cloud" tutorials stop at a single curl that returns one completion. That is a demo, not a service. In this guide, we build the real thing: a production-grade, OpenAI-compatible LLM API on a DigitalOcean GPU Droplet, served by SGLang - one of the fastest open-source inference engines - quantized to fit and fly, hardened behind Nginx, supervised by systemd, and benchmarked so you know exactly how many tokens per second and dollars per million tokens you are getting.

This is the sequel to our CPU guide,
Run Llama3.1-8B on a DigitalOcean CPU droplet.
That post shows you how to get an LLM running cheaply on CPU. This one shows you
how to serve one to real users on a GPU — and how to keep the bill small.

What you will build

Client ──HTTPS──▶ Nginx (TLS + API key) ──▶ SGLang server (:30000) ──▶ GPU
                  reverse proxy, rate cap      OpenAI-compatible      Llama-3.1-8B (fp8)

By the end you will have:

• An OpenAI-compatible endpoint (/v1/chat/completions) you can point any OpenAI SDK at.
• FP8 weight + KV-cache quantization so an 8B model fits comfortably and serves faster.
• Nginx in front for TLS and a shared-secret API key.
• A systemd unit so the server survives reboots and crashes.
• Real throughput / latency numbers from sglang.bench_serving.
• A cost model and an on-demand workflow that cuts the GPU bill dramatically.

Why SGLang (and not just vLLM)

SGLang is a high-throughput serving engine built around RadixAttention (automatic prefix-cache reuse across requests), continuous batching, and an OpenAI-compatible front end. For workloads with shared prefixes—system prompts, few-shot examples, RAG templates, agent loops—that prefix reuse is a large, free speedup. It ships a launch_server entry point, native FP8 support, and a built-in benchmarking tool, which is exactly the toolkit we need for a clean production deploy.

Prerequisites

• A DigitalOcean account with GPU Droplets enabled.
• A Hugging Face account + access token (Llama models are gated — accept the license once).
• Basic comfort with SSH and the Linux shell.
• Optional but recommended: doctl, the DO CLI.

Step 1: Pick and create the GPU Droplet

For an 8B model quantized to FP8 you need roughly 10 GB of VRAM for weights plus KV cache and activation headroom. A single mid-range data-center GPU (24 GB class or above) is plenty and leaves room for healthy batch sizes. Rule of thumb:

Create it from the console (Create → GPU Droplet → pick an AI/ML-ready Ubuntu image
with NVIDIA drivers preinstalled), or with doctl:

doctl compute droplet create sglang-serve \
  --region nyc2 \
  --size <gpu-droplet-size-slug> \
  --image <gpu-ready-ubuntu-image-slug> \
  --ssh-keys <your-fingerprint> \
  --wait

Use doctl compute size list and doctl compute image list --public to find the
exact GPU size and AI/ML image slugs available in your region. Prefer an image that
already bundles the NVIDIA driver + container toolkit so you skip driver setup.

SSH in:

ssh root@<droplet-ip>

Step 2: Verify the GPU

Before anything else, confirm the driver sees the card:

nvidia-smi

You should see the GPU, its total memory, and a driver/CUDA version. If this fails, you booted a non-GPU image or the driver is missing — recreate the droplet from an AI/ML image rather than fighting driver installs.

Step 3: Install SGLang

The most reproducible path is the official Docker image (it pins CUDA and kernels for you). Make sure the NVIDIA Container Toolkit is present (docker run --gpus all must work):

docker run --gpus all --shm-size 32g --ipc=host \

-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env HF_TOKEN=<your_hf_token> \
lmsysorg/sglang:latest

python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 --port 30000

Prefer a bare-metal install? SGLang recommends uv:

pip install --upgrade pip
pip install uv
uv pip install "sglang[all]"

Either way, the first launch downloads the model weights from Hugging Face into the cache volume that is why we mount ~/.cache/huggingface (it means restarts and rebuilds don't re-download tens of GB).

Step 4: Launch the server and smoke-test it

Start the OpenAI-compatible server:

python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 --port 30000

From another shell, hit the OpenAI-compatible endpoint:

curl http://localhost:30000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": [
            {
                "role": "user",
                "content": "Explain RadixAttention in one sentence."
            }
        ],
        "max_tokens": 128
    }'

Because it speaks the OpenAI protocol, any OpenAI SDK works by just changing the base URL:

from openai import OpenAI

client = OpenAI(
    base_url="http://<droplet-ip>:30000/v1",
    api_key="sk-local"
)

r = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    # Remaining lines are cut off in the image.
)

    messages=[{"role": "user", "content": "Say hi in 3 languages."}],
)
print(r.choices[0].message.content)

Step 5: Quantize to fit and go faster

FP8 quantization roughly halves memory versus FP16 and increases throughput on modern GPUs, usually with negligible quality loss for chat workloads. Two independent knobs matter:

python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --quantization fp8 \
    --kv-cache-dtype fp8_e4m3 \
    --context-length 8192 \
    --mem-fraction-static 0.85 \
    --host 0.0.0.0 --port 30000

• --quantization fp8 : FP8 weights (halves weight memory vs FP16).
• --kv-cache-dtype fp8_e4m3 : FP8 KV cache, so you fit far more concurrent tokens/requests in the same VRAM (bigger effective batch = higher throughput).
• --context-length : cap it to what you actually need; KV cache scales with it.
• --mem-fraction-static : how much VRAM to reserve for the KV pool. Nudge it down if you hit out-of-memory at load, up if you have spare VRAM and want bigger batches.

Accuracy note. FP8 is safe for most chat/RAG use, but validate on your prompts before trusting it in production. If you need pre-quantized checkpoints (AWQ/GPTQ/FP8), point --model-path at the quantized repo directly.

Step 6 — Put it behind Nginx (TLS + API key)

Never expose the raw inference port to the internet. Bind SGLang to localhost and let Nginx terminate TLS and enforce a shared-secret key. Install and configure:

apt-get update && apt-get install -y nginx

/etc/nginx/sites-available/sglang:

server {
    listen 80;
    server_name llm.yourdomain.com;

    location /v1/ {
        # simple shared-secret gate

        if ($http_authorization != "Bearer YOUR_SECRET_KEY") { return 401; }

        proxy_pass         http://127.0.0.1:30000;
        proxy_set_header   Host $host;
        proxy_read_timeout 300s;     # long generations
        proxy_buffering    off;      # stream tokens as they arrive
    }
}

Enable it and add HTTPS with Let's Encrypt:

ln -s /etc/nginx/sites-available/sglang /etc/nginx/sites-enabled/
nginx -t && systemctl reload nginx
apt-get install -y certbot python3-certbot-nginx
certbot --nginx -d llm.yourdomain.com

Now relaunch SGLang bound to localhost only: change --host 0.0.0.0 to --host 127.0.0.1 and close port 30000 in the DO firewall.

Step 7: Run it as a service (systemd)

So the server restarts on crash and comes back after a reboot. /etc/systemd/system/sglang.service:

[Unit]
Description=SGLang LLM server
After=network-online.target
Wants=network-online.target

[Service]
Environment=HF_TOKEN=your_hf_token
ExecStart=/usr/bin/python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --quantization fp8 --kv-cache-dtype fp8_e4m3 \
    --context-length 8192 --mem-fraction-static 0.85 \
    --host 127.0.0.1 --port 30000
Restart=always
RestartSec=5
User=root

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now sglang

journalctl -u sglang -f # watch the logs

Step 8: Benchmark it

Numbers you didn't measure are numbers you don't have. SGLang ships a load generator:

python3 -m sglang.bench_serving \
    --backend sglang \
    --num-prompts 200 \
    --request-rate 8 \
    --random-input-len 1024 \
    --random-output-len 256

Read the report for output tokens/sec (throughput), TTFT (time to first token – your users' perceived latency), and end-to-end latency per request. Sweep --request-rate up until TTFT degrades to find the load your droplet can sustain.

Record results in a table like this (example shape – replace with your measured values):

Always publish the machine context alongside the numbers – GPU model, driver/CUDA version, model, quantization, input/output lengths. A tokens/sec figure without that context is meaningless.

The cost models and a smarter way to run it

The headline question is dollars per million tokens. Compute it from what you measured:

$ / 1M output tokens =
(droplet $/hour ÷ output_tokens_per_sec ÷ 3600) × 1,000,000

So if a droplet costs $H/hour and sustains T output tokens/sec:

• Example: $H = 1.50/hr, T = 2000 tok/s
  → 1.5 / 2000 / 3600 × 1e6 ≈ $0.21 per million output tokens.
• Plug in your measured T and the current droplet price for a real figure.

The catch with any always-on GPU: you pay for idle time. A box that serves traffic 8 hours a day but bills 24 wastes two-thirds of its cost. Here is the workflow most tutorials never mention:

Treat the GPU Droplet as a disposable, snapshot-backed rig

1. Bake a snapshot once everything works (drivers + SGLang image + a Volume holding the model weights). Weights on a Block Storage Volume mean you never re-download the model when you recreate the droplet.
2. Spin up on demand for batch jobs, evaluations, or business hours; destroy it when idle. You are billed by the hour, so this is a direct, linear saving.
3. Automate it with doctl so it's one command (or a cron / CI trigger):

# start for the workday
doctl compute droplet create sglang-serve --image <your-snapshot-id> \
  --size <gpu-size> --region nyc2 --ssh-keys <fp> --wait

# ...run your workload...

# stop paying when done
doctl compute droplet delete sglang-serve --force

1. Keep a reserved IP so the endpoint address is stable across recreations, and put the model weights on a Volume so recreation is seconds, not a re-download.

This "destroy-when-idle" pattern turns a fixed monthly GPU bill into a variable, usage-shaped one often a 50–80% reduction for bursty or business-hours workloads, while still giving you full control of the model and data (unlike a hosted API).

Troubleshooting

• CUDA out of memory at load → lower --mem-fraction-static, reduce --context-length, or ensure FP8 flags are actually applied.
• nvidia-smi not found → wrong image; recreate from an AI/ML GPU image.
• docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]] → NVIDIA Container Toolkit not installed/registered.
• 401 from the API → the Authorization: Bearer ... header must match the Nginx gate.
• Model download hangs / 403 → accept the model license on Hugging Face and pass a valid HF_TOKEN.
• Throughput lower than expected → raise --request-rate in the benchmark (you may be under-loading), enable FP8 KV cache for bigger batches, and confirm requests share prefixes so RadixAttention can help.

Wrap-up

You now have a real LLM service on DigitalOcean: quantized with SGLang, fronted by Nginx with TLS and an API key, kept alive by systemd, measured with a proper load test, and most importantly run in a cost-shaped way with snapshots and on-demand lifecycle instead of an always-on bill.

Keep going:

• Prequel on CPU: [Run Llama3.1-8B on a DigitalOcean CPU droplet](https://iq.opengenus.org/llama-on-digitalocean-cpu-droplet)
• Learn to measure GPUs properly in our hands-on AI Performance Engineering course (roofline, tokens/sec, quantization).
• SGLang documentation for tensor parallelism, speculative decoding, and structured output.