Guide to GPU-accelerated ML tasks on Intel Jasper Lake (11th gen) processors with integrated GPU

[thariq-shanavas]

There appear to be some shenanigans with the LP mode on Intel Jasper Lake processors that prevent ML tasks from running on the GPU. The issues seem to be fixed by loading the GuC and HuC firmwares. This guide will walk you through installing the proprietary i915 firmware, GuC (graphics micro controller) firmware and HuC (HEVC micro controller) firmware.

This guide was tested on Debian 12 with the Linux 6.1 kernel, but things may change in the future.

From Intel [1],

GuC is designed to perform graphics workload scheduling on the various graphics
parallel engines. In this scheduling model, the host software submits work through one
of the 256 graphics doorbells. This invokes the scheduling operation on the
appropriate graphics engine. Scheduling operations include determining which
workload to run next, submitting a workload to a command streamer, pre-empting
existing workloads running on an engine, monitoring progress, and notifying the host
software when work is done.
HuC is designed to offload some of the media functions from the CPU to GPU. These
include bitrate control and header parsing. For example, in the case of bitrate control,
the driver invokes the HuC in the beginning of each frame encoding pass. The encode
bitrate is adjusted by the calculation from HuC. Both the HuC hardware and the encode
hardcode reside in the GPU. Using the HuC will save unnecessary CPU-GPU
synchronization.

This may not be enabled by default in many Linux distros. Before enabling the GuC, my ML container would frequently crash with an out of GPU resources error, copied below.

2024-07-28 22:21:14.703408648 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running OpenVINO-EP-subgraph_3 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_3_0' Status Message: /onnxruntime/onnxruntime/core/providers/openvino/ov_interface.cc:243 void onnxruntime::openvino_ep::OVInferRequest::WaitRequest() [OpenVINO-EP]  Wait Model Failed: Exception from src/inference/src/cpp/infer_request.cpp:245:
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_engine.cpp:185:
[GPU] out of GPU resources

[07/28/24 22:21:14] ERROR    Exception in ASGI application

                             ╭─────── Traceback (most recent call last) ───────╮
                             │ /usr/src/app/main.py:152 in predict             │
                             │                                                 │
                             │   149 │   │   inputs = text                     │
                             │   150 │   else:                                 │
                             │   151 │   │   raise HTTPException(400, "Either  │
                             │ ❱ 152 │   response = await run_inference(inputs │
                             │   153 │   return ORJSONResponse(response)       │
                             │   154                                           │
                             │   155                                           │
                             │                                                 │
                             │ /usr/src/app/main.py:177 in run_inference       │
                             │                                                 │
                             │   174 │   without_deps, with_deps = entries     │
                             │   175 │   await asyncio.gather(*[_run_inference │
                             │   176 │   if with_deps:                         │
                             │ ❱ 177 │   │   await asyncio.gather(*[_run_infer │
                             │   178 │   if isinstance(payload, Image):        │
                             │   179 │   │   response["imageHeight"], response │
                             │   180                                           │
                             │                                                 │
                             │ /usr/src/app/main.py:170 in _run_inference      │
                             │                                                 │
                             │   167 │   │   │   │   message = f"Task {entry[' │
                             │       output of {dep}"                          │
                             │   168 │   │   │   │   raise HTTPException(400,  │
                             │   169 │   │   model = await load(model)         │
                             │ ❱ 170 │   │   output = await run(model.predict, │
                             │   171 │   │   outputs[model.identity] = output  │
                             │   172 │   │   response[entry["task"]] = output  │
                             │   173                                           │
                             │                                                 │
                             │ /usr/src/app/main.py:188 in run                 │
                             │                                                 │
                             │   185 │   if thread_pool is None:               │
                             │   186 │   │   return func(*args, **kwargs)      │
                             │   187 │   partial_func = partial(func, *args, * │
                             │ ❱ 188 │   return await asyncio.get_running_loop │
                             │   189                                           │
                             │   190                                           │
                             │   191 async def load(model: InferenceModel) ->  │
                             │                                                 │
                             │ /usr/local/lib/python3.11/concurrent/futures/th │
                             │ read.py:58 in run                               │
                             │                                                 │
                             │ /usr/src/app/models/base.py:60 in predict       │
                             │                                                 │
                             │    57 │   │   self.load()                       │
                             │    58 │   │   if model_kwargs:                  │
                             │    59 │   │   │   self.configure(**model_kwargs │
                             │ ❱  60 │   │   return self._predict(*inputs, **m │
                             │    61 │                                         │
                             │    62 │   @abstractmethod                       │
                             │    63 │   def _predict(self, *inputs: Any, **mo │
                             │                                                 │
                             │ /usr/src/app/models/facial_recognition/recognit │
                             │ ion.py:46 in _predict                           │
                             │                                                 │
                             │   43 │   │   │   return []                      │
                             │   44 │   │   inputs = decode_cv2(inputs)        │
                             │   45 │   │   cropped_faces = self._crop(inputs, │
                             │ ❱ 46 │   │   embeddings = self._predict_batch(c │
                             │      self._predict_single(cropped_faces)        │
                             │   47 │   │   return self.postprocess(faces, emb │
                             │   48 │                                          │
                             │   49 │   def _predict_batch(self, cropped_faces │
                             │      NDArray[np.float32]:                       │
                             │                                                 │
                             │ /usr/src/app/models/facial_recognition/recognit │
                             │ ion.py:50 in _predict_batch                     │
                             │                                                 │
                             │   47 │   │   return self.postprocess(faces, emb │
                             │   48 │                                          │
                             │   49 │   def _predict_batch(self, cropped_faces │
                             │      NDArray[np.float32]:                       │
                             │ ❱ 50 │   │   embeddings: NDArray[np.float32] =  │
                             │   51 │   │   return embeddings                  │
                             │   52 │                                          │
                             │   53 │   def _predict_single(self, cropped_face │
                             │      NDArray[np.float32]:                       │
                             │                                                 │
                             │ /opt/venv/lib/python3.11/site-packages/insightf │
                             │ ace/model_zoo/arcface_onnx.py:84 in get_feat    │
                             │                                                 │
                             │   81 │   │                                      │
                             │   82 │   │   blob = cv2.dnn.blobFromImages(imgs │
                             │   83 │   │   │   │   │   │   │   │   │     (sel │
                             │      self.input_mean), swapRB=True)             │
                             │ ❱ 84 │   │   net_out = self.session.run(self.ou │
                             │   85 │   │   return net_out                     │
                             │   86 │                                          │
                             │   87 │   def forward(self, batch_data):         │
                             │                                                 │
                             │ /usr/src/app/sessions/ort.py:49 in run          │
                             │                                                 │
                             │    46 │   │   input_feed: dict[str, NDArray[np. │
                             │    47 │   │   run_options: Any = None,          │
                             │    48 │   ) -> list[NDArray[np.float32]]:       │
                             │ ❱  49 │   │   outputs: list[NDArray[np.float32] │
                             │       run_options)                              │
                             │    50 │   │   return outputs                    │
                             │    51 │                                         │
                             │    52 │   @property                             │
                             │                                                 │
                             │ /opt/venv/lib/python3.11/site-packages/onnxrunt │
                             │ ime/capi/onnxruntime_inference_collection.py:22 │
                             │ 0 in run                                        │
                             │                                                 │
                             │    217 │   │   if not output_names:             │
                             │    218 │   │   │   output_names = [output.name  │
                             │    219 │   │   try:                             │
                             │ ❱  220 │   │   │   return self._sess.run(output │
                             │    221 │   │   except C.EPFail as err:          │
                             │    222 │   │   │   if self._enable_fallback:    │
                             │    223 │   │   │   │   print(f"EP Error: {err!s │
                             ╰─────────────────────────────────────────────────╯
                             Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero
                             status code returned while running
                             OpenVINO-EP-subgraph_3 node.
                             Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgrap
                             h_3_0' Status Message:
                             /onnxruntime/onnxruntime/core/providers/openvino/ov
                             _interface.cc:243 void
                             onnxruntime::openvino_ep::OVInferRequest::WaitReque
                             st() [OpenVINO-EP]  Wait Model Failed: Exception
                             from src/inference/src/cpp/infer_request.cpp:245:
                             Exception from
                             src/plugins/intel_gpu/src/runtime/ocl/ocl_engine.cp
                             p:185:
                             [GPU] out of GPU resources

The rest of this guide follows instructions from Jellyfin [2].

1. Install the latest linux firmware packages on the host system.
   On Debian:
   sudo apt update && sudo apt install -y firmware-linux-nonfree
   On Ubuntu:
   sudo apt update && sudo apt install -y linux-firmware
   On Arch Linux:
   sudo pacman -Syu linux-firmware
   Or from the Linux repo directly:

  cd ~/
  git clone --depth=1 https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
  sudo mkdir -p /usr/lib/firmware
  sudo cp -r linux-firmware/i915 /usr/lib/firmware

2. Add the required i915 kernel parameter on the host system to enable loading GuC and HuC firmware:

sudo mkdir -p /etc/modprobe.d
sudo sh -c "echo 'options i915 enable_guc=2' >> /etc/modprobe.d/i915.conf"

3. Update the initramfs and grub. The commands varies between distros.
   On Debian & Ubuntu:
   sudo update-initramfs -u && sudo update-grub
   On Arch Linux:
   sudo mkinitcpio -P && sudo update-grub

4. Reboot the system
   sudo reboot

5. Check the GuC & HuC status with the following commands, make sure there is no FAIL or ERROR in the outputs.

  sudo dmesg | grep i915
  sudo cat /sys/kernel/debug/dri/0/gt/uc/guc_info
  sudo cat /sys/kernel/debug/dri/0/gt/uc/huc_info

If you get a No such file or directory error when running the last two commands, try querying a dri device with a different number, for example 1:

sudo cat /sys/kernel/debug/dri/1/gt/uc/guc_info
sudo cat /sys/kernel/debug/dri/1/gt/uc/huc_info

6. Change your docker compose file to use the OpenVINO backend
   https://immich.app/docs/features/ml-hardware-acceleration

7. Clear the model cache
   docker volume rm immich-app_model-cache or docker volume rm model-cache depending on your setup.
   You can find the name of the docker volume containing the ML cache using docker volume ls

8. Restart Immich

And that's it! Many thanks to the Immich machine learning wizard @mertalev for help figuring this out.

[1] https://www.intel.com/content/www/us/en/content-details/609249/enabling-the-guc-huc-firmware-for-linux-on-new-intel-gpu-platforms.html
[2] https://jellyfin.org/docs/general/administration/hardware-acceleration/intel/#configure-and-verify-lp-mode-on-linux

[Target0998]

Thank you for your guide, it helped me a lot!
I was able to get it running.

What I do not understand, is that it stops after a couple minutes or so.

I'm running:

The outputs from:

sudo dmesg | grep i915
sudo cat /sys/kernel/debug/dri/0/gt0/uc/guc_info
sudo cat /sys/kernel/debug/dri/0/gt0/uc/huc_info

[    9.110117] i915 0000:00:02.0: [drm] VT-d active for gfx access
[    9.113539] i915 0000:00:02.0: vgaarb: deactivate vga console
[    9.113646] i915 0000:00:02.0: [drm] Using Transparent Hugepages
[    9.120041] i915 0000:00:02.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[    9.122872] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/icl_dmc_ver1_09.bin (v1.9)
[    9.124519] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_ops [i915])
[    9.137067] i915 0000:00:02.0: [drm] GT0: GuC firmware i915/ehl_guc_70.1.1.bin version 70.1.1
[    9.137075] i915 0000:00:02.0: [drm] GT0: HuC firmware i915/ehl_huc_9.0.0.bin version 9.0.0
[    9.151258] i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads
[    9.151267] i915 0000:00:02.0: [drm] GT0: GUC: submission disabled
[    9.151269] i915 0000:00:02.0: [drm] GT0: GUC: SLPC disabled
[    9.154254] [drm] Initialized i915 1.6.0 20230929 for 0000:00:02.0 on minor 0
[    9.156992] i915 display info: display version: 11
[    9.156996] i915 display info: cursor_needs_physical: no
[    9.156997] i915 display info: has_cdclk_crawl: no
[    9.156998] i915 display info: has_cdclk_squash: no
[    9.157000] i915 display info: has_ddi: yes
[    9.157001] i915 display info: has_dp_mst: yes
[    9.157002] i915 display info: has_dsb: no
[    9.157003] i915 display info: has_fpga_dbg: yes
[    9.157004] i915 display info: has_gmch: no
[    9.157005] i915 display info: has_hotplug: yes
[    9.157006] i915 display info: has_hti: no
[    9.157007] i915 display info: has_ipc: yes
[    9.157008] i915 display info: has_overlay: no
[    9.157009] i915 display info: has_psr: yes
[    9.157010] i915 display info: has_psr_hw_tracking: yes
[    9.157011] i915 display info: overlay_needs_physical: no
[    9.157012] i915 display info: supports_tv: no
[    9.157013] i915 display info: has_hdcp: yes
[    9.157014] i915 display info: has_dmc: yes
[    9.157015] i915 display info: has_dsc: yes
[   11.065953] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[   11.066110] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[   12.971967] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[   14.890948] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[  211.261548] i915 0000:00:02.0: Using 39-bit DMA addresses
GuC firmware: i915/ehl_guc_70.1.1.bin
        status: RUNNING
        version: found 70.1.1
        uCode: 274112 bytes
        RSA: 256 bytes
GuC status 0x8003f0ec:
        Bootrom status = 0x76
        uKernel status = 0xf0
        MIA Core status = 0x3
Scratch registers:
         0:     0x0
         1:     0x1773d7
         2:     0x44000
         3:     0x4000
         4:     0x40
         5:     0x514
         6:     0x4e610001
         7:     0x0
         8:     0x0
         9:     0x0
        10:     0x0
        11:     0x0
        12:     0x0
        13:     0x0
        14:     0x0
        15:     0x0

GuC logging stats:
        Relay full count: 0
        DEBUG:  flush count          0, overflow count          0
        CRASH:  flush count          0, overflow count          0
        CAPTURE:        flush count          0, overflow count          0
HuC firmware: i915/ehl_huc_9.0.0.bin
        status: RUNNING
        version: found 9.0.0
        uCode: 498496 bytes
        RSA: 256 bytes
HuC status: 0x0007c001

I do not understand, because I use this setup for my jellyfin setup, and there it works.
Can someone help me get this running?
Or at least point me into the right direction?

[thariq-shanavas]

@Target0998 I will preface this by saying I don't quite understand why enabling the GuC and HuC works, just that it did on my system. I therefore have very limited ability to help you troubleshoot your setup.

That said, there are three options for the enable_guc setting in /etc/modprobe.d/i915.conf

Try all three of these settings by editing your /etc/modprobe.d/i915.conf file - I found that enable_guc=2 was the only setting that worked without crashing on my system, hopefully one of the other options may work for you. Good luck!