fix: Run Bedrock calls in executor for async

[lou-k]

Wraps synchronous calls in an executor for async support in Bedrock evals.

The current code makes a synchronous call in an async function. As a result, calling run_evals:

1. Doesn't actually parallelize while using bedrock, resulting in very low throughput
2. Doesn't properly display the progress bar since all calls are made before the gather. This confuses the user and makes them think something is wrong.

Proper async support in boto3 has been asked for for years, so I don't think we can count on it any time soon. Wrapping in an executor ties up some threads on I/O, but still results in significantly higher throughput, which I think eval users (including myself) require.

[anticorrelator]

@lou-k Thanks so much for the contribution! Because we don't have comprehensive testing around our model wrappers before we can merge this I was hoping we could check (or even explicitly test) the following cases:

• Does llm_classify successfully parallelize calls to boto in a script via a sync entry point?
• Does llm_classify successfully parallelize calls to boto when we need a nested event loop (e.g. in a notebook environment)
• Do errors raised inside the boto call get properly propagated outside of the thread? (If we mock the boto call and have it return an error, will that error show up in the llm_classify "exceptions" column?)
• Does this interact properly with the openinference Bedrock auto-instrumentation?

If we can confirm all of this behavior is preserved this seems like a great idea!

[lou-k]

@anticorrelator thanks for the feedback! I'll run these tests this week and share results.

[lou-k]

@anticorrelator :

Does llm_classify successfully parallelize calls to boto in a script via a sync entry point?

Yes, confirmed locally by running the toxicity eval in a script

Does llm_classify successfully parallelize calls to boto when we need a nested event loop (e.g. in a notebook environment)

Yes, confirmed locally by running the toxicity benchmark notebook

Do errors raised inside the boto call get properly propagated outside of the thread? (If we mock the boto call and have it return an error, will that error show up in the llm_classify "exceptions" column?)

I've added a unit test that confirms errors are throln from the async call

Does this interact properly with the openinference Bedrock auto-instrumentation?

Yes, but it's worth calling out that calls to claude2 don't work properly with Bedrock's instrumentation right now since the code assumes a prompt is in the request body, and this BedrockModel class uses the messages api for claude2

I noticed tox is failing but I can't seem to find the error message -- can you see it?

[anticorrelator]

@lou-k Thanks for checking! Let me try running tox locally on your branch

[anticorrelator]

@lou-k looks like the import block needs formatting

tests/phoenix/evals/models/test_bedrock.py:1:1: I001 [*] Import block is un-sorted or un-formatted
Found 1 error.

If you run the ruff formatter everything should be good to go: ruff format .; ruff check . --fix;

[anticorrelator]

Thank you so much!

[lou-k]

Ahh thanks! I'm new to ruff -- I'll fix it up and squash the commits

[lou-k]

I believe this is ready to merge :highfive: