Python: Bug: Python Semantic-Kernel 1.10.0 SDK is not compatible with the latest pydantic 2.9.2

[HuskyDanny]

Describe the bug
I got the error "TypeError: cannot pickle '_thread.RLock' object" when working with semantic-kernel==1.10.0 and pydantic==2.9.2.

Then I downgrade them to semantic-kernel==1.7.0 and pydantic==2.8.2 pydantic-settings==2.4.0 pydantic_core==2.20.1, everything works

Screenshots

Platform

• OS: Windows
• IDE: VS Code
• Language: Python
• Source: [e.g. NuGet package version 0.1.0, pip package version 0.1.0, main branch of repository]

Additional context
Full Error:
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/site-packages/semantic_kernel/init.py", line 3, in
from semantic_kernel.kernel import Kernel
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/site-packages/semantic_kernel/kernel.py", line 31, in
from semantic_kernel.functions.function_result import FunctionResult
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/site-packages/semantic_kernel/functions/init.py", line 5, in
from semantic_kernel.functions.kernel_function import KernelFunction
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/site-packages/semantic_kernel/functions/kernel_function.py", line 55, in
class KernelFunction(KernelBaseModel):
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 205, in new
complete_model_class(
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 567, in complete_model_class
generate_pydantic_signature(init=cls.init, fields=cls.model_fields, config_wrapper=config_wrapper),
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/site-packages/pydantic/_internal/_signature.py", line 159, in generate_pydantic_signature
merged_params = _generate_signature_parameters(init, fields, config_wrapper)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/site-packages/pydantic/_internal/_signature.py", line 115, in _generate_signature_parameters
kwargs = {} if field.is_required() else {'default': field.get_default(call_default_factory=False)}
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/site-packages/pydantic/fields.py", line 554, in get_default
return _utils.smart_deepcopy(self.default)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/site-packages/pydantic/_internal/_utils.py", line 318, in smart_deepcopy
return deepcopy(obj) # slowest way when we actually might need a deepcopy
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 211, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 211, in
y = [deepcopy(a, memo) for a in x]
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/allenpan/anaconda3/envs/provisionTest/lib/python3.10/copy.py", line 161, in deepcopy
rv = reductor(4)
TypeError: cannot pickle '_thread.RLock' object

[moonbox3]

Hi @HuskyDanny, thanks for filing the issue. Can you give us a bit more context as to what you're doing to get this error? I have not seen this before. During all check-ins to main we're running unit test and integration tests (that also exercise concept samples, learn site samples and notebooks) and this error hasn't occurred. Any other info you can provide to help us repro it would be great. Thanks.

[JTremb]

I have the same issue, but on my side it seems to be related to opentelemetry

in my code if I switch from semantic-kernel==1.7.0 to any version from semantic-kernel==1.8.0 and later (no other change) , and I get the same error as shown above.

but in my code if I comment those lines

from opentelemetry import metrics
meter_provider = MeterProvider( ...)
metrics.set_meter_provider(meter_provider )

it work with semantic-kernel==1.8.0

[HuskyDanny]

Hi @moonbox3, I was basically trying to spin up my fast api app locally, and it broke immediately. Switching the dependent package versions resolve this for now, but I am concerned if not resolving this issue, updating to further semantic kernel version would be blocked

[moonbox3]

This should be fixed as of #9292. Please ping back on this if you continue to experience any issues. Thanks.