Ollama in-cluster

Schema: un initContainer (model-fetcher) scarica il modello dal registry OCI in una PVC una sola volta; Ollama lo serve su un Service ClusterIP. Nessun accesso a internet.

Manifest

cat > ollama-ai-llm.yaml <<'EOF2'
apiVersion: v1
kind: Namespace
metadata:
  name: ai-llm
  labels:
    purpose: ai-llm
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-models
  namespace: ai-llm
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 40Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ai-llm
  labels: { app: ollama }
spec:
  replicas: 1
  strategy: { type: Recreate }
  selector:
    matchLabels: { app: ollama }
  template:
    metadata:
      labels: { app: ollama }
    spec:
      imagePullSecrets:
        - name: registry-internal
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001        # remove on OpenShift (restricted-v2 assigns the UID)
        fsGroup: 1001          # remove on OpenShift
        seccompProfile: { type: RuntimeDefault }
      initContainers:
        - name: model-fetcher
          image: registry.internal/ai/model-fetcher:1.0
          env:
            - { name: MODEL_REF, value: "registry.internal/ai-models/qwen2.5-14b-instruct:ollama-q4_k_m" }
            - { name: DEST, value: /data }
            - { name: REGISTRY_CONFIG, value: /auth/.dockerconfigjson }
            - { name: CA_FILE, value: /ca/ca.crt }
          securityContext:
            allowPrivilegeEscalation: false
            capabilities: { drop: [ALL] }
          volumeMounts:
            - { name: models, mountPath: /data }
            - { name: registry-auth, mountPath: /auth, readOnly: true }
            - { name: registry-ca, mountPath: /ca, readOnly: true }
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { memory: 1Gi }
      containers:
        - name: ollama
          image: registry.internal/ai/ollama:<tag>
          env:
            - { name: HOME, value: /data }
            - { name: OLLAMA_MODELS, value: /data/models }
            - { name: OLLAMA_HOST, value: "0.0.0.0:11434" }
            - { name: OLLAMA_KEEP_ALIVE, value: "24h" }
            - { name: OLLAMA_MAX_LOADED_MODELS, value: "1" }
            - { name: OLLAMA_NUM_PARALLEL, value: "2" }
            - { name: OLLAMA_CONTEXT_LENGTH, value: "16384" }
            - { name: OLLAMA_FLASH_ATTENTION, value: "1" }
            - { name: OLLAMA_KV_CACHE_TYPE, value: "q8_0" }
          ports:
            - { name: http, containerPort: 11434 }
          securityContext:
            allowPrivilegeEscalation: false
            capabilities: { drop: [ALL] }
          readinessProbe:
            httpGet: { path: /api/tags, port: http }
            periodSeconds: 10
          livenessProbe:
            httpGet: { path: /api/version, port: http }
            initialDelaySeconds: 30
            periodSeconds: 30
          resources:
            requests: { cpu: "8", memory: 16Gi }
            limits: { memory: 20Gi }
            # GPU nodes:
            # limits: { memory: 20Gi, nvidia.com/gpu: 1 }
          volumeMounts:
            - { name: models, mountPath: /data }
      volumes:
        - name: models
          persistentVolumeClaim: { claimName: ollama-models }
        - name: registry-auth
          secret:
            secretName: registry-internal
            items: [{ key: .dockerconfigjson, path: .dockerconfigjson }]
        - name: registry-ca
          configMap: { name: registry-ca }
---
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: ai-llm
spec:
  selector: { app: ollama }
  ports:
    - { name: http, port: 11434, targetPort: http }
EOF2

Il Secret registry-internal e la ConfigMap registry-ca sono creati in Modello OCI.

kubectl apply -f ollama-ai-llm.yaml

kubectl -n ai-llm logs deploy/ollama -c model-fetcher -f

kubectl -n ai-llm rollout status deploy/ollama --timeout=20m

Verifica

kubectl -n ai-llm exec deploy/ollama -- ollama list

kubectl -n ai-llm run curl --rm -it --restart=Never --image=registry.internal/ai/curl:8 -- curl -s http://ollama.ai-llm.svc:11434/v1/models

Warm-up (carica il modello in memoria prima del primo utilizzo):

kubectl -n ai-llm exec deploy/ollama -- ollama run qwen2.5:14b-instruct "ready?"

kubectl -n ai-llm exec deploy/ollama -- ollama ps

GPU

Con NVIDIA GPU Operator installato, decommenta nvidia.com/gpu: 1 nei limits e aggiungi, se i nodi GPU sono tainted:

      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      nodeSelector:
        nvidia.com/gpu.present: "true"

kubectl -n ai-llm exec deploy/ollama -- nvidia-smi

Aggiornare il modello

kubectl -n ai-llm set env deploy/ollama -c model-fetcher MODEL_REF=registry.internal/ai-models/qwen2.5-14b-instruct:ollama-q4_k_m-v2

Il vecchio modello resta nella PVC; rimuovilo quando il nuovo è validato:

kubectl -n ai-llm exec deploy/ollama -- ollama rm <old-model>

Modello derivato k8s-sre in-cluster

kubectl -n ai-llm exec deploy/ollama -- sh -c 'printf "FROM qwen2.5:14b-instruct\nPARAMETER num_ctx 16384\nPARAMETER temperature 0.1\nSYSTEM You are a senior Kubernetes SRE. Give root cause, evidence and exact kubectl command. Never invent resource names.\n" > /data/Modelfile.k8s-sre && ollama create k8s-sre -f /data/Modelfile.k8s-sre'