vLLM su GPU

vLLM serve modelli in formato Hugging Face (safetensors) con batching continuo: molte richieste parallele senza degrado lineare. È la scelta quando più tool o più persone usano l’LLM contemporaneamente. Richiede GPU.

Dimensionamento GPU

Modello	Formato	VRAM pesi	GPU minima
Qwen2.5-7B-Instruct	BF16	~15 GB	24 GB (L4, A10)
Qwen2.5-14B-Instruct-AWQ	AWQ 4-bit	~10 GB	24 GB (L4, A10)
Qwen2.5-14B-Instruct	BF16	~29 GB	48 GB (L40S, A6000)
Qwen2.5-32B-Instruct-AWQ	AWQ 4-bit	~19 GB	48 GB

Il resto della VRAM va alla KV cache: più ne resta, più richieste parallele e contesto.

Manifest

cat > vllm-ai-llm.yaml <<'EOF2'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vllm-models
  namespace: ai-llm
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 60Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
  namespace: ai-llm
  labels: { app: vllm }
spec:
  replicas: 1
  strategy: { type: Recreate }
  selector:
    matchLabels: { app: vllm }
  template:
    metadata:
      labels: { app: vllm }
    spec:
      imagePullSecrets:
        - name: registry-internal
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
        - { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }
      initContainers:
        - name: model-fetcher
          image: registry.internal/ai/model-fetcher:1.0
          env:
            - { name: MODEL_REF, value: "registry.internal/ai-models/qwen2.5-14b-instruct:hf-awq" }
            - { name: DEST, value: /models }
            - { name: REGISTRY_CONFIG, value: /auth/.dockerconfigjson }
            - { name: CA_FILE, value: /ca/ca.crt }
          volumeMounts:
            - { name: models, mountPath: /models }
            - { name: registry-auth, mountPath: /auth, readOnly: true }
            - { name: registry-ca, mountPath: /ca, readOnly: true }
      containers:
        - name: vllm
          image: registry.internal/ai/vllm-openai:<tag>
          args:
            - --model=/models/qwen2.5-14b-instruct-awq
            - --served-model-name=qwen2.5-14b-instruct
            - --quantization=awq
            - --max-model-len=16384
            - --gpu-memory-utilization=0.90
            - --enable-auto-tool-choice
            - --tool-call-parser=hermes
            - --port=8000
          env:
            - { name: HF_HUB_OFFLINE, value: "1" }
            - { name: TRANSFORMERS_OFFLINE, value: "1" }
            - { name: VLLM_NO_USAGE_STATS, value: "1" }
            - { name: HOME, value: /tmp }
          ports:
            - { name: http, containerPort: 8000 }
          readinessProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 10
          startupProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 10
            failureThreshold: 60
          resources:
            requests: { cpu: "4", memory: 24Gi }
            limits: { memory: 32Gi, nvidia.com/gpu: 1 }
          volumeMounts:
            - { name: models, mountPath: /models }
            - { name: shm, mountPath: /dev/shm }
      volumes:
        - name: models
          persistentVolumeClaim: { claimName: vllm-models }
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 8Gi }
        - name: registry-auth
          secret:
            secretName: registry-internal
            items: [{ key: .dockerconfigjson, path: .dockerconfigjson }]
        - name: registry-ca
          configMap: { name: registry-ca }
---
apiVersion: v1
kind: Service
metadata:
  name: vllm
  namespace: ai-llm
spec:
  selector: { app: vllm }
  ports:
    - { name: http, port: 8000, targetPort: http }
EOF2

--tool-call-parser=hermes è quello corretto per la famiglia Qwen2.5; per altri modelli verifica il parser nella documentazione vLLM.

kubectl apply -f vllm-ai-llm.yaml

kubectl -n ai-llm logs deploy/vllm -f

Verifica

kubectl -n ai-llm run curl --rm -it --restart=Never --image=registry.internal/ai/curl:8 -- curl -s http://vllm.ai-llm.svc:8000/v1/models

kubectl -n ai-llm run curl --rm -it --restart=Never --image=registry.internal/ai/curl:8 -- curl -s http://vllm.ai-llm.svc:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen2.5-14b-instruct","messages":[{"role":"user","content":"ping"}]}'

Metriche Prometheus (token/s, richieste in coda, KV cache):

kubectl -n ai-llm exec deploy/vllm -- curl -s localhost:8000/metrics | grep -E '^vllm:(num_requests|gpu_cache_usage|generation_tokens)' | head

Puntare i tool a vLLM

Tool	Impostazione
k8sgpt	backend: localai, baseUrl: http://vllm.ai-llm.svc:8000/v1, model: qwen2.5-14b-instruct
HolmesGPT	OPENAI_API_BASE=http://vllm.ai-llm.svc:8000/v1, OPENAI_API_KEY=dummy, MODEL=openai/qwen2.5-14b-instruct
kubectl-ai	provider OpenAI-compatible verso lo stesso endpoint