-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
.Net: Replace SK's IAIServiceProvider with IServiceProvider (#3714)
Closes #3694 Closes #3693 Closes #3659 Closes #3571 Closes #3191 Closes #2463 (This replaces #3676, including that as a first commit. That resiliency functionality is no longer necessary after this change, as developers can simply use libraries like Microsoft.Extensions.Http.Resilience to achieve the desired support.) SK has had a custom IAIServiceProvider interface that's represented a provider of AI related services. Kernel has then wrapped one of these, but also wrapped other individual non-AI services, like ILoggerFactory and HttpClient. This leads to a variety of problems, such as it being more complicated to construct a Kernel instance, it not integrating as nicely with a broader dependency injection system, and not being as future proof because there isn't a good way to flow additional non-AI-related services through the Kernel to anywhere that might want access. On the DI front, it's also more difficult to integrate because all of the KernelBuilder extension methods are specific to KernelBuilder, and even though they're creating services, the helpers aren't usable with IServiceCollection and thus with the rest of the DI system; a developer needs to go directly to the underlying types and register those directly. There are also a handful of features that are of little value and in some cases exposed but not actually implemented, such as setAsDefault and alsoAsTextCompletion. And other cases where there's something valuable but missing, such as the ability to create certain kinds of Azure-related service instances with an existing OpenAIClient. This PR attempts to address all of these issues and set up SK to evolve more easily: - Kernel now wraps an IServiceProvider, and all services (AI and non-AI) are retrieved from it. This has a variety of benefits, including future-proofing for wanting to flow more state around, better integration with other systems (e.g. Microsoft.Extensions.DependencyInjection), simpler construction, etc. - KernelBuilder still has dedicated WithXx methods for first-class features, but they're all layered on top of a new ConfigureServices method, which lets you add anything to an IServiceCollection. Because everything is based on IServiceCollection/IServiceProvider, dependencies are automatically satisfied from the container. For example, for services that accept an optional HttpClient, if one isn't specified to the WithXx method, the system will pull one out of the DI container if it can, which means all existing support for adding configured HttpClient instances to an IServiceCollection immediately apply to Kernel{Builder} as well. - KernelBuilder also exposes a ConfigurePlugins method, largely for parity with ConfigureServices, as fundmantally a Kernel is just the combination of a collection of services and a collection of plugins (plus some additional transient data, event handlers, etc.) This also integrates with IServiceProvider, with one passed into the callback, so that if the plugin itself needs access to something like logging, it can simply get it from the container. These ConfigureServices and ConfigurePlugins methods can be called any number of times. - KernelBuilder is now just a simplified, opinionated way to achieve something you can achieve in two other ways: construct the Kernel directly, or construct the Kernel via DI. For the latter, you can simply add Kernel as a DI resource, and it'll automatically be composed out of what's available. For example, if in ASP.NET you AddAzureOpenChatCompletion, AddLogging, AddHttp, etc., and then ask for a Kernel to be injected, that Kernel will be constructed from the IServiceProvider that contains all of those other resources, and thus anyone using the Kernel will get that logger, will get that HttpClient, etc.; the HttpClient will also be used by the Azure OpenAI chat completion. @matthewbolanos, @markwallace-microsoft, @SergeyMenshykh, please review from an SK perspective to let me know if you agree with the direction, whether there are specific things you'd like to see changed, etc. There are some straggling things we should still do after this (like add a few more extension methods for adding plugins to a plugin collection, adding more tests, revamping XML comments, cleaning up more code, etc.), but this is a sweeping change and I wanted to get it in. @eerhardt, @halter73, I'd appreciate it if you could review this from a DI perspective, e.g. are there variations in the patterns I should be following, any best-practices I'm violating, whether I'm using singleton when I should be using transient, etc. etc. Note that I ran into one issue as part of this: the Microsoft.Extensions.DependencyInjection support for keyed services doesn't let you enumerate all services for a particular T, regardless of key. That's something SK depends on. To workaround that, KernelBuilder tracks all of the type/keys and injects a dictionary as a service with that information; then places the Kernel needs it, it can query for that dictionary and use it. This means that a Kernel constructed with KernelBuilder will fully support keys, whereas a Kernel constructed with DI or directly won't work as nicely with named services. My hope is that M.E.DI can fix this asap (offline conversation) in a way that will allow SK to upgrade to a patched version, delete its hack, and have those broader scenarios "just work". In the meantime, though, KernelBuilder behaves as desired. I also stopped short of adding Add{Azure}OpenAIXx methods that resolve the OpenAIClient from the service provider. That'll be an important thing to do once there's an Aspire component for Azure.AI.OpenAI, but in the meantime, I didn't want there to be an attractive Add{Azure}OpenAIXx method that looked like you didn't need to supply the necessary information even though you do. --------- Co-authored-by: SergeyMenshykh <sergemenshikh@gmail.com>
- Loading branch information
1 parent
b7f9b1d
commit 73afd47
Showing
190 changed files
with
2,214 additions
and
5,303 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
175 changes: 22 additions & 153 deletions
175
dotnet/samples/KernelSyntaxExamples/Example08_RetryHandler.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,177 +1,46 @@ | ||
| // Copyright (c) Microsoft. All rights reserved. | ||
|
|
||
| using System; | ||
| using System.IO; | ||
| using System.Net; | ||
| using System.Net.Http; | ||
| using System.Threading; | ||
| using System.Threading.Tasks; | ||
| using Microsoft.Extensions.DependencyInjection; | ||
| using Microsoft.Extensions.Http.Resilience; | ||
| using Microsoft.Extensions.Logging; | ||
| using Microsoft.SemanticKernel; | ||
| using Microsoft.SemanticKernel.Http; | ||
| using Microsoft.SemanticKernel.Plugins.Core; | ||
| using Microsoft.SemanticKernel.Reliability.Basic; | ||
| using Polly; | ||
| using RepoUtils; | ||
|
|
||
| // ReSharper disable once InconsistentNaming | ||
| #pragma warning disable CA1031 // Do not catch general exception types | ||
| #pragma warning disable CA2000 // Dispose objects before losing scope | ||
|
|
||
| public static class Example08_RetryHandler | ||
| { | ||
| public static async Task RunAsync() | ||
| { | ||
| await DefaultNoRetryAsync(); | ||
|
|
||
| await ReliabilityBasicExtensionAsync(); | ||
|
|
||
| await ReliabilityPollyExtensionAsync(); | ||
|
|
||
| await CustomHandlerAsync(); | ||
| } | ||
|
|
||
| private static async Task DefaultNoRetryAsync() | ||
| { | ||
| InfoLogger.Logger.LogInformation("============================== Kernel default behavior: No Retry =============================="); | ||
| var kernel = InitializeKernelBuilder() | ||
| .Build(); | ||
|
|
||
| await ImportAndExecutePluginAsync(kernel); | ||
| } | ||
|
|
||
| private static async Task ReliabilityBasicExtensionAsync() | ||
| { | ||
| InfoLogger.Logger.LogInformation("============================== Using Reliability.Basic extension =============================="); | ||
| var retryConfig = new BasicRetryConfig | ||
| // Create a Kernel with the HttpClient | ||
| var kernel = new KernelBuilder().ConfigureServices(c => | ||
| { | ||
| MaxRetryCount = 3, | ||
| UseExponentialBackoff = true, | ||
| }; | ||
| retryConfig.RetryableStatusCodes.Add(HttpStatusCode.Unauthorized); | ||
|
|
||
| var kernel = InitializeKernelBuilder() | ||
| .WithRetryBasic(retryConfig) | ||
| .Build(); | ||
|
|
||
| await ImportAndExecutePluginAsync(kernel); | ||
| } | ||
|
|
||
| private static async Task ReliabilityPollyExtensionAsync() | ||
| { | ||
| InfoLogger.Logger.LogInformation("============================== Using Reliability.Polly extension =============================="); | ||
| var kernel = InitializeKernelBuilder() | ||
| .WithRetryPolly(GetPollyPolicy(InfoLogger.LoggerFactory)) | ||
| .Build(); | ||
|
|
||
| await ImportAndExecutePluginAsync(kernel); | ||
| } | ||
|
|
||
| private static async Task CustomHandlerAsync() | ||
| { | ||
| InfoLogger.Logger.LogInformation("============================== Using a Custom Http Handler =============================="); | ||
| var kernel = InitializeKernelBuilder() | ||
| .WithHttpHandlerFactory(new MyCustomHandlerFactory()) | ||
| .Build(); | ||
|
|
||
| await ImportAndExecutePluginAsync(kernel); | ||
| } | ||
|
|
||
| private static KernelBuilder InitializeKernelBuilder() | ||
| { | ||
| return new KernelBuilder() | ||
| .WithLoggerFactory(InfoLogger.LoggerFactory) | ||
| // OpenAI settings - you can set the OpenAI.ApiKey to an invalid value to see the retry policy in play | ||
| .WithOpenAIChatCompletionService(TestConfiguration.OpenAI.ChatModelId, "BAD_KEY"); | ||
| } | ||
|
|
||
| private static AsyncPolicy<HttpResponseMessage> GetPollyPolicy(ILoggerFactory? logger) | ||
| { | ||
| // Handle 429 and 401 errors | ||
| // Typically 401 would not be something we retry but for demonstration | ||
| // purposes we are doing so as it's easy to trigger when using an invalid key. | ||
| const int TooManyRequests = 429; | ||
| const int Unauthorized = 401; | ||
|
|
||
| return Policy | ||
| .HandleResult<HttpResponseMessage>(response => | ||
| (int)response.StatusCode is TooManyRequests or Unauthorized) | ||
| .WaitAndRetryAsync(new[] | ||
| c.AddLogging(c => c.AddConsole().SetMinimumLevel(LogLevel.Information)); | ||
| c.ConfigureHttpClientDefaults(c => | ||
| { | ||
| // Use a standard resiliency policy, augmented to retry on 401 Unauthorized for this example | ||
| c.AddStandardResilienceHandler().Configure(o => | ||
| { | ||
| TimeSpan.FromSeconds(2), | ||
| TimeSpan.FromSeconds(4), | ||
| TimeSpan.FromSeconds(8) | ||
| }, | ||
| (outcome, timespan, retryCount, _) | ||
| => InfoLogger.Logger.LogWarning("Error executing action [attempt {RetryCount} of 3], pausing {PausingMilliseconds}ms. Outcome: {StatusCode}", | ||
| retryCount, | ||
| timespan.TotalMilliseconds, | ||
| outcome.Result.StatusCode)); | ||
| } | ||
|
|
||
| private static async Task ImportAndExecutePluginAsync(Kernel kernel) | ||
| { | ||
| // Load semantic plugin defined with prompt templates | ||
| string folder = RepoFiles.SamplePluginsPath(); | ||
|
|
||
| kernel.ImportPluginFromObject<TimePlugin>(); | ||
|
|
||
| var qaPlugin = kernel.ImportPluginFromPromptDirectory(Path.Combine(folder, "QAPlugin")); | ||
| o.Retry.ShouldHandle = args => ValueTask.FromResult(args.Outcome.Result?.StatusCode is HttpStatusCode.Unauthorized); | ||
| }); | ||
| }); | ||
| c.AddOpenAIChatCompletion("gpt-4", "BAD_KEY"); // OpenAI settings - you can set the OpenAI.ApiKey to an invalid value to see the retry policy in play | ||
| }).Build(); | ||
|
|
||
| var question = "How popular is Polly library?"; | ||
| var logger = kernel.GetService<ILoggerFactory>().CreateLogger(typeof(Example08_RetryHandler)); | ||
|
|
||
| InfoLogger.Logger.LogInformation("Question: {0}", question); | ||
| // To see the retry policy in play, you can set the OpenAI.ApiKey to an invalid value | ||
| #pragma warning disable CA1031 // Do not catch general exception types | ||
| const string Question = "How popular is Polly library?"; | ||
| logger.LogInformation("Question: {Question}", Question); | ||
| try | ||
| { | ||
| var answer = await kernel.InvokeAsync(qaPlugin["Question"], question); | ||
| InfoLogger.Logger.LogInformation("Answer: {0}", answer.GetValue<string>()); | ||
| logger.LogInformation("Answer: {Result}", await kernel.InvokePromptAsync(Question)); | ||
| } | ||
| catch (Exception ex) | ||
| { | ||
| InfoLogger.Logger.LogInformation("Error: {0}", ex.Message); | ||
| } | ||
| #pragma warning restore CA1031 // Do not catch general exception types | ||
| } | ||
|
|
||
| // Basic custom retry handler factory | ||
| public sealed class MyCustomHandlerFactory : HttpHandlerFactory<MyCustomHandler> | ||
| { | ||
| } | ||
|
|
||
| // Basic custom empty retry handler | ||
| public sealed class MyCustomHandler : DelegatingHandler | ||
| { | ||
| public MyCustomHandler(ILoggerFactory loggerFactory) | ||
| { | ||
| } | ||
|
|
||
| protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) | ||
| { | ||
| // Your custom http handling implementation | ||
| return Task.FromResult(new HttpResponseMessage(HttpStatusCode.BadRequest) | ||
| { | ||
| Content = new StringContent("My custom bad request override") | ||
| }); | ||
| } | ||
| } | ||
|
|
||
| private static class InfoLogger | ||
| { | ||
| internal static ILogger Logger => LoggerFactory.CreateLogger("Example08_RetryHandler"); | ||
| internal static ILoggerFactory LoggerFactory => s_loggerFactory.Value; | ||
| private static readonly Lazy<ILoggerFactory> s_loggerFactory = new(LogBuilder); | ||
|
|
||
| private static ILoggerFactory LogBuilder() | ||
| { | ||
| return Microsoft.Extensions.Logging.LoggerFactory.Create(builder => | ||
| { | ||
| builder.SetMinimumLevel(LogLevel.Information); | ||
| builder.AddFilter("Microsoft", LogLevel.Information); | ||
| builder.AddFilter("Microsoft.SemanticKernel", LogLevel.Critical); | ||
| builder.AddFilter("Microsoft.SemanticKernel.Reliability", LogLevel.Information); | ||
| builder.AddFilter("System", LogLevel.Information); | ||
|
|
||
| builder.AddConsole(); | ||
| }); | ||
| logger.LogInformation("Error: {Message}", ex.Message); | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.