[BUG] ECONNABORTED and timeouts when indexing

[bestrocker221]

The bug

Added a local library and started the indexing..
Takes forever, and it actually crashes due to javascript heap out of memory (same as in here #3844.)

The machine is modestly powerful, Ryzen 5 (16 cores) + 46GB RAM.

Even after restarting the docker compose, I get these errors almost immediately.

immich_microservices     | [Nest] 7  - 10/29/2023, 4:27:09 PM     LOG [MetadataService] Initialized local reverse geocoder with cities500
immich_microservices     | [Nest] 7  - 10/29/2023, 4:27:09 PM     LOG [SearchService] Running bootstrap
immich_microservices     | Request #1698596829552: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1698596829552: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1698596829552: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1698596829552: Sleeping for 4s and then retrying request...
immich_microservices     | [Nest] 7  - 10/29/2023, 4:27:35 PM   ERROR [JobService] Unable to run job handler (sidecar/sidecar-discovery): Error: Connection terminated due to connection timeout
immich_microservices     | [Nest] 7  - 10/29/2023, 4:27:35 PM   ERROR [JobService] Error: Connection terminated due to connection timeout
immich_microservices     |     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
immich_microservices     |     at Object.onceWrapper (node:events:628:28)
immich_microservices     |     at Connection.emit (node:events:514:28)
immich_microservices     |     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
immich_microservices     |     at Socket.emit (node:events:514:28)
immich_microservices     |     at TCP.<anonymous> (node:net:337:12)
immich_microservices     | [Nest] 7  - 10/29/2023, 4:27:35 PM   ERROR [JobService] Object

Then other related errors (many of them)

immich_microservices     | [Nest] 7  - 10/29/2023, 4:30:45 PM   ERROR [JobService] Unable to run job handler (sidecar/sidecar-discovery): Error: Connection terminated due to connection timeout
immich_microservices     | [Nest] 7  - 10/29/2023, 4:30:45 PM   ERROR [JobService] Error: Connection terminated due to connection timeout
immich_microservices     |     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
immich_microservices     |     at Object.onceWrapper (node:events:628:28)
immich_microservices     |     at Connection.emit (node:events:514:28)
immich_microservices     |     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
immich_microservices     |     at Socket.emit (node:events:514:28)
immich_microservices     |     at TCP.<anonymous> (node:net:337:12)
immich_microservices     | [Nest] 7  - 10/29/2023, 4:30:45 PM   ERROR [JobService] Object:

I tried lowering the concurrency level to 3 for all generations, still same problem.
CPU utilization is not really high at all, seems unused for the most part.
The only container using a core or two is the microservices one, unless doing ffmpeg transcoding.

The OS that Immich Server is running on

Debian 11

Version of Immich Server

1.83

Version of Immich Mobile App

1.83

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${MYLIB_LOCATION}:/usr/src/app/mylibrary:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${MYLIB_LOCATION}:/usr/src/app/mylibrary:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - ./model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - ./tsdata:/data
    restart: always
  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - ./pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

Your .env content

UPLOAD_LOCATION=XXX
MYLIB_LOCATION=YYY

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

### added by me
NODE_OPTIONS="--max-old-space-size=8192"
TYPESENSE_API_KEY=blah-bliggedy-blah
PUBLIC_LOGIN_PAGE_MESSAGE="Your own private photo library."
###

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=some-random-text
DB_PASSWORD=postgres

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1.Start immich
2.Add external folder as library
3.Start indexing
...

Additional information

No response