.Net: feature/llm openapi payload

[baywet]

• feat: adds basic infrastructure for proxy operation runner
• feat: adds chat client to proxy
• feat: adds chat client to the OpenApi extensions methods

[SergeyMenshykh]

Before starting to consider a dedicated runner that would use an AI model to create a payload dynamically, I would suggest trying/considering one of the "cheap" options available in SK today:

1. Use the Kernel.InvokePromptAsync method. No additional code is required; just import the OpenAPI document with dynamic payload construction disabled to the kernel, register an AI service on the kernel, and call the method with all details in the prompt. SK will send the prompt to the AI service with the imported operations metadata (which includes the JSON schema for the payload) and handle the function call (invoke the requested function) from the model. This sample shows how it can be done: InvokeOpenApiFunctionWithPayloadProvidedByCallerAsync.
2. Decorate/wrap an OpenAPI plugin functions. The decorator will "intercept" a function call, use an AI model to construct a payload based on the original/decorated function's schema and available context/arguments, and invoke the decorated function with the payload and other required arguments. This sample showing how to decorate an SK function - How to transform a KernelFunction

[baywet]

Thanks for the input.
I believe 2 is effectively what I'm trying to implement. From our earlier discussion I didn't get the fact that you were suggesting an attribute based implementation/AOP approach.
With this approach, how would one decide/configure which "chat service" to use to generate the payload?

[baywet]

The other challenge I can see here with transformations is they run once during the setup of the function, whereas the payload generation needs to run once per function call.

[SergeyMenshykh]

Thanks for the input. I believe 2 is effectively what I'm trying to implement. From our earlier discussion I didn't get the fact that you were suggesting an attribute based implementation/AOP approach.

No need for attribute based/AOP wrappers. Each operation just needs to be wrapped in a delegate that will be invoked every time the original function would be called:

With this approach, how would one decide/configure which "chat service" to use to generate the payload?

SK can inject kernel instance to each function. Just add a parameter of type Kernel to your function and current instance of the kernel will be injected. In the example above, the delegate's first parameter is kernel. Having kernel in your function allows the function to access all services via Kernel.Services property of IServiceProvider type registered on kernel.

[baywet]

Thank you for the additional information.

And according to you, why is a proxy implementation for the runner more "costly"?

[SergeyMenshykh]

Thank you for the additional information.

And according to you, why is a proxy implementation for the runner more "costly"?

Because a) SK can do it today out of the box for the majority of scenarios (see option 1 in the first comment), and if further customization is needed, it's supported as well (see option 2 ), and b) the new AI-powered runner will require maintenance and will eventually need to be made public and configurable, either fully or partially, to accommodate various customization requests from customers. This, in turn, will increase SK's public API surface and decrease development speed because we don't want to introduce breaking changes to the public API surface. Therefore, the current preference is to leverage the existing options until there are strong signals in favor of the runner.

UPD: I would also suggest creating an ADR at https://github.com/microsoft/semantic-kernel/tree/main/docs/decisions that outlines the new proposed design, highlights its pros and cons, and compares it to the suggested and available today options.