fix: cache boto3 swf clients #434
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Do not instantiate a swf client on each and every object creation. Use a per-thread cache as clients are not threadsafe. Signed-off-by: Yves Bastide <yves@botify.com>
We've got a generic cache architecture, let's use it for s3 too 🙂 Signed-off-by: Yves Bastide <yves@botify.com>
Signed-off-by: Yves Bastide <yves@botify.com>
ewjoachim
reviewed
Oct 11, 2023
simpleflow/boto3_utils.py
Outdated
| from __future__ import annotations | ||
|
|
||
| import hashlib | ||
| from threading import local |
There was a problem hiding this comment.
Please use ContextVars instead https://docs.python.org/3/library/contextvars.html
SofienBenAyed
approved these changes
Oct 11, 2023
simpleflow/boto3_utils.py
Outdated
Comment on lines
21
to
22
| boto3_clients = _client_var.get("boto3_clients") | ||
| if boto3_clients is None: |
There was a problem hiding this comment.
In [3]: _client_var = ContextVar("boto3_clients", default=None)
In [4]: boto3_clients = _client_var.get("boto3_clients")
In [5]: boto3_clients
Out[5]: 'boto3_clients'
By default, boto3_clients is initialized to the string "boto3_clients"
* the "local data" variable must be global, obviously * replace threading.local with contextvars.ContextVar: the future's here! Signed-off-by: Yves Bastide <yves@botify.com>
ybastide
force-pushed
the
reported-bug/DATA-18922/oom-on-simpleflow
branch
from
ce81e03 to
e31bb92
Compare
SofienBenAyed
approved these changes
Oct 11, 2023
ewjoachim
approved these changes
Oct 11, 2023
simpleflow/boto3_utils.py
Outdated
| key = hashlib.sha1(json_dumps(d).encode()).hexdigest() | ||
| boto3_clients = _client_var.get() | ||
| if boto3_clients is None: | ||
| boto3_clients = {} |
There was a problem hiding this comment.
this line would then be removed
ewjoachim
approved these changes
Oct 11, 2023
|
Good nitpicks, I'll apply them and make a rc1 🙂 |
ybastide
commented
Oct 11, 2023
Thank you Joachim! Co-authored-by: Joachim Jablon <ewjoachim@gmail.com>
SofienBenAyed
approved these changes
Oct 11, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Yves Bastide yves@botify.com