Huggingface inference API not working with openai_compatible

[bassamsdata]

I'm trying to integrate the Hugging Face Inference API. I tested it with the openai and openai_compatible adapters, but it didn't work. The challenge lies in its unique URL structure, where the model is part of the URL itself. This makes switching between multiple models tricky, as changing the repo/model (e.g., meta-llama/Llama-3.2-3B-Instruct) requires altering the URL as well.

curl 'https://api-inference.huggingface.co/models/meta-llama/Llama-3.2-3B-Instruct/v1/chat/completions' \
--header "Authorization: Bearer {YOUR_HF_TOKEN}" \
--header 'Content-Type: application/json' \
--data '{
    "model": "meta-llama/Llama-3.2-3B-Instruct",
    "messages": [
    { "role": "user", "content": "what is the Capital of France?" },
    { "role": "assistant", "content": "The capital of France is Paris." },
    { "role": "user", "content": "what is the Capital of UK?" },
    { "role": "assistant", "content": "The capital of the United Kingdom is London." },
    { "role": "user", "content": "what is the Capital of Canada?" },
    { "role": "assistant", "content": "The capital of Canada is Ottawa." }
],
    "temperature": 0.5,
    "max_tokens": 2048,
    "top_p": 0.7,
    "stream": true
}'

If anyone can assist, I’d greatly appreciate it. If there’s currently no solution, I’d be happy to add a Hugging Face adapter to CodeCompanion, given its extensive model library (complete with tests, of course).

Thank you

edit: add code curl

[olimorris]

What's your adapter currently look like? I would have thought the openai_compatible would work just fine. Something like:

require("codecompanion").setup({
  adapters = {
    hugging_face = function()
      return require("codecompanion.adapters").extend("openai_compatible", {
        url = "${url}/${model}/${chat_url}",
        env = {
          url = "https://api-inference.huggingface.co/models",
          model = "schema.model.default",
          chat_url = "v1/chat/completions",
          api_key = "YOUR_HF_KEY",
        },
        schema = {
          model = {
            default = "meta-llama/Llama-3.2-3B-Instruct"
          },
        },
      })
  }
)}

[olimorris]

Could be worth trying this the OpenAI adapter. I don't have any way of testing the openai_compatible adapter (it was a community pull request) and I am concerned that it doesn't leverage all of the OpenAI Adapters handlers.

[bassamsdata]

Thank you for the detailed information and support.

Initially, because didn't have time, I sent everything to the LLM (sonnet 3.5) , but unfortunately, but it couldn't find the issue with multiple prompts (though the trace log was incredibly detailed and impressive).

Later in the evening, I decided to dig into the source code of the OpenAI adapter schema and compare it against the parameters supported by Huggingface, analyzing them one by one.

Finally, I managed to get it working perfectly. The issues boiled down to the following points:

1. The first two parameters caused unexpected issues (though I’m unsure why).
2. The last two parameters were not supported by Huggingface.

For reference, I used the OpenAI adapter as a base since it was easier/shorter to read, and even extended it to support Groq with some additional tweaks.

params.presence_penalty = nil
params.frequency_penalty = nil
params.logit_bias = nil
params.user = nil

Final one working.

          huggingface = function()
            return require("codecompanion.adapters").extend("openai", {
              name = "huggingface",
              url = "${url}/models/${model}/v1/chat/completions",
              env = {
                api_key = huggingface_api_key,
                url = "https://api-inference.huggingface.co",
                model = "schema.model.default",
              },
              headers = {
                ["Content-Type"] = "application/json",
                Authorization = "Bearer ${api_key}",
              },
              handlers = {
                form_parameters = function(self, params, messages)
                  params.presence_penalty = nil
                  params.frequency_penalty = nil
                  params.logit_bias = nil
                  params.user = nil

                  return params
                end,

              },
              schema = {
                model = {
                  default = "mistralai/Mistral-Nemo-Instruct-2407",
                  choices = {
                    "meta-llama/Llama-3.2-1B-Instruct",
                    "Qwen/Qwen2.5-72B-Instruct",
                    "google/gemma-2-2b-it",
                    "mistralai/Mistral-Nemo-Instruct-2407",
                  },
                },
                temperature = {
                  default = 0.5,
                },
                top_p = {
                  default = 0.7,
                },
                max_tokens = {
                  default = 2048,
                },
              },
            })
          end,

thank you

[bassamsdata]

One unrelated question, if you don't mind:

If I want to develop functionality to send an image to llm and save an image coming from the llm as well, where would be the best place to start looking in the source code?

[olimorris]

That adapter is a thing of beauty! Would you consider making a PR and adding this to the plugin? You could pretty much copy the xAI and OpenAI ones.

If I want to develop functionality to send an image to llm and save an image coming from the llm as well, where would be the best place to start looking in the source code?

Vision capabilities are something I've thought about since the plugin's inception. Infact, I even have a base64 encoder in the utils folder. The main reason I hadn't pursued it further was because I couldn't workout how to keep the UI clean as we'd need to parse an image path, encode it and add it to the message stack. This was all before I'd had slash commands implemented and this notion of references in the chat buffer.

You're welcome to poke around the chat/init.lua file. It would be a combination of understanding the message stack in the chat object and how we manipulate messages for each adapter. Slash Commands would also be a useful start point too if we went down that route. I would definitely welcome us riffing on a PR to add vision support.

[bassamsdata]

That adapter is a thing of beauty! Would you consider making a PR and adding this to the plugin? You could pretty much copy the xAI and OpenAI ones.

I'll work on creating one as soon as I can.

I would definitely welcome us riffing on a PR to add vision support.

Thanks for the details! I'll review the internals and determine the simplest approach to implement this.