Machine Learning scaling - New replicas do not receive any jobs

[jstebenne]

The bug

I am experimenting with using k3s Horizontal Pod Autoscalers, and I've set it up with a machine learning deployment. I've observed the same issue when scaling manually.

When new replicas of the machine learning image are created, they will stay idle until I pause the job inside the administration page, and then unpause them. I would expect the new replicas to pick up pending jobs instead of having to do this manual action.

The OS that Immich Server is running on

K3S v1.29.2

Version of Immich Server

v1.123.0

Version of Immich Mobile App

n/a

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

Kubernetes Definitions:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: immich-machine-learning
  namespace: immich
  labels:
    app.kubernetes.io/instance: immich
    app.kubernetes.io/name: machine-learning
spec:
  revisionHistoryLimit: 3
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: machine-learning
      app.kubernetes.io/instance: immich
  template:
    metadata:
      labels:
        app.kubernetes.io/name: machine-learning
        app.kubernetes.io/instance: immich
    spec:
      serviceAccountName: default
      automountServiceAccountToken: true
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: machine-learning
            app.kubernetes.io/instance: immich
      containers:
        - name: immich-machine-learning
          image: ghcr.io/immich-app/immich-machine-learning:v1.123.0
          imagePullPolicy: IfNotPresent
          env:
            - name: DB_DATABASE_NAME
              value: [REDACTED]
            - name: DB_HOSTNAME
              value: immich-postgresql
            - name: DB_PASSWORD
              value: [REDACTED]
            - name: DB_USERNAME
              value: [REDACTED]
            - name: IMMICH_MACHINE_LEARNING_URL
              value: http://immich-machine-learning:3003
            - name: REDIS_HOSTNAME
              value: immich-redis-master
            - name: TRANSFORMERS_CACHE
              value: /cache
          ports:
            - name: http
              containerPort: 3003
              protocol: TCP
          resources:
            limits:
              cpu: 4
              memory: 12Gi
            requests:
              cpu: 0.01
              memory: 1Gi
          volumeMounts:
            - name: cache
              mountPath: /cache
          livenessProbe:
            failureThreshold: 10
            httpGet:
              path: /ping
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /ping
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 1
      volumes:
        - name: cache
          persistentVolumeClaim:
            claimName: immich-machine-learning-cache

---

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  namespace: immich
  name: machine-learning-scaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: immich-machine-learning
  minReplicas: 1
  maxReplicas: 4
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 3000

Your .env content

n/a

Reproduction steps

1. Assuming a working Immich setup with plenty of photos (17k+ in my case) and one machine-learning replica
2. Go to the Jobs page and refresh Face Detection
3. Observe the machine-learning pod reach its resource limit
4. Observe the autoscaler create additional machine-learning pods
5. The CPU usage of new pods stay at 0

Relevant log output

Logs of new machine-learning pod:

[12/21/24 00:59:32] INFO     Starting gunicorn 23.0.0                           
[12/21/24 00:59:32] INFO     Listening at: http://[::]:3003 (9)                 
[12/21/24 00:59:32] INFO     Using worker: app.config.CustomUvicornWorker       
[12/21/24 00:59:32] INFO     Booting worker with pid: 10                        
[12/21/24 00:59:37] INFO     Started server process [10]                        
[12/21/24 00:59:37] INFO     Waiting for application startup.                   
[12/21/24 00:59:37] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                     
[12/21/24 00:59:37] INFO     Initialized request thread pool with 4 threads.    
[12/21/24 00:59:37] INFO     Application startup complete.

Additional information

No response

[mertalev]

I think this is a limitation of autoscaling. You can try setting MACHINE_LEARNING_HTTP_KEEPALIVE_TIMEOUT_S=0 to see if that fixes your issue.