InvalidClientIdError thrown when the token expires

[evelinagkougklia]

Describe the bug
I am using OAuth2 in impersonation mode with a FaultTolerance retry policy. When the token expires, an InvalidClientIdError exception is thrown. The re-authentication seems to happen on its own, it's just weird that the exception is thrown. It's also not clear to me if the request that caused the token to appear as expired is retried after the successful reauthentication, or if it's on us to implement retries.

To Reproduce

from time import sleep

from exchangelib import OAuth2Credentials, Identity, Configuration, OAUTH2, FaultTolerance, Account


credentials = OAuth2Credentials(
    client_id='CLIENT_ID',
    client_secret='CLIENT_SECRET',
    tenant_id='TENANT_ID',
    identity=Identity(primary_smtp_address='test@test.com')
)
config = Configuration(server='outlook.office365.com', credentials=credentials, auth_type=OAUTH2,
                       retry_policy=FaultTolerance(max_wait=3600))
account = Account('test@test.com', config=config, autodiscover=False)

# run a continuous script for a while until the 1-hour mark, when the token expires
while True:
    # dummy script so that there's something being filtered
    messages = account.inbox.all()
    for m in messages[10]:
        print(m.id)
    sleep(10)

Expected behavior
Either a clarification on if there's something wrong with my configuration / setup or how to safely recover from this error.

Log output

[urllib3.connectionpool] [DEBUG] Resetting dropped connection: login.microsoftonline.com
[urllib3.connectionpool] [DEBUG] https://login.microsoftonline.com:443 "POST /xxxxxxxxxxx/oauth2/v2.0/token HTTP/11" 400 497
Response XML: None
Request XML: b'<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types"><s:Header><t:RequestServerVersion Version="Exchange2016"/><t:ExchangeImpersonation><t:ConnectingSID><t:PrimarySmtpAddress>test@test.com</t:PrimarySmtpAddress></t:ConnectingSID></t:ExchangeImpersonation><t:TimeZoneContext><t:TimeZoneDefinition Id="UTC"/></t:TimeZoneContext></s:Header><s:Body><m:FindItem Traversal="Shallow"><m:ItemShape><t:BaseShape>IdOnly</t:BaseShape></m:ItemShape><m:IndexedPageItemView MaxEntriesReturned="1000" Offset="0" BasePoint="Beginning"/><m:Restriction><t:IsGreaterThan><t:FieldURI FieldURI="item:DateTimeReceived"/><t:FieldURIOrConstant><t:Constant Value="2024-12-18T12:55:22Z"/></t:FieldURIOrConstant></t:IsGreaterThan></m:Restriction><m:ParentFolderIds><t:FolderId Id="xxxxx" ChangeKey="xxxxxxx"/></m:ParentFolderIds></m:FindItem></s:Body></s:Envelope>'
Response headers: None
Request headers: {'X-AnchorMailbox': 'test@test.com'}
Status code: None
Response time: None
Streaming: False
HTTP adapter: <requests.adapters.HTTPAdapter object at 0x7fc7af37b490>
URL: https://outlook.office365.com/EWS/Exchange.asmx
Auth type: <requests_oauthlib.oauth2_auth.OAuth2 object at 0x7fc7afebcc50>
Thread: 123345123141234
Session: 12345
Timeout: 120
[exchangelib.util] [ERROR] InvalidClientIdError: (invalid_request) AADSTS900144: The request body must contain the following parameter: 'refresh_token'. Trace ID: xxxxxx Correlation ID: xxxxxxx Timestamp: 2024-12-18 13:55:28Z
[urllib3.connectionpool] [DEBUG] Starting new HTTPS connection (1): login.microsoftonline.com:443
[urllib3.connectionpool] [DEBUG] https://login.microsoftonline.com:443 "POST /xxxxxxxxxxx/oauth2/v2.0/token HTTP/11" 200 1761

Additional context
exchangelib v.5.4.2
python v.3.11.11

[ecederstrand]

It's only a log message, not an exception. So you can assume that everything is working correctly. I can see if I can convert the log message to something less scary.

Unless you actually got a stack trace and forgot to attach it here?

[evelinagkougklia]

Sorry, yes, I have a stack trace. I was just passing the exceptions until I thought to ask here.

Traceback (most recent call last):
  File "/opt/.venv/lib/python3.11/site-packages/requests_oauthlib/oauth2_session.py", line 528, in request
    url, headers, data = self._client.add_token(
                         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 217, in add_token
    raise TokenExpiredError()
oauthlib.oauth2.rfc6749.errors.TokenExpiredError: (token_expired) 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/handler/core/email_handler.py", line 103, in process_emails
    if messages.count() > 0:
       ^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/exchangelib/queryset.py", line 549, in count
    return len(list(new_qs.__iter__()))
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/exchangelib/queryset.py", line 270, in __iter__
    yield from self._format_items(items=self._query(), return_format=self.return_format)
  File "/opt/.venv/lib/python3.11/site-packages/exchangelib/queryset.py", line 345, in _item_yielder
    for i in iterable:
  File "/opt/.venv/lib/python3.11/site-packages/exchangelib/folders/collections.py", line 211, in find_items
    yield from FindItem(account=self.account, page_size=page_size).call(
  File "/opt/.venv/lib/python3.11/site-packages/exchangelib/services/common.py", line 216, in _elems_to_objs
    for elem in elems:
  File "/opt/.venv/lib/python3.11/site-packages/exchangelib/services/common.py", line 801, in _paged_call
    pages = self._get_pages(payload_func, kwargs, len(paging_infos))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/exchangelib/services/common.py", line 898, in _get_pages
    page_elems = list(self._get_elements(payload=payload))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/exchangelib/services/common.py", line 300, in _get_elements
    yield from self._response_generator(payload=payload)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/exchangelib/services/common.py", line 784, in _response_generator
    response = self._get_response_xml(payload=payload)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/exchangelib/services/common.py", line 396, in _get_response_xml
    r = self._get_response(payload=payload, api_version=api_version)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/exchangelib/services/common.py", line 347, in _get_response
    r, session = post_ratelimited(
                 ^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/exchangelib/util.py", line 825, in post_ratelimited
    r = session.post(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/requests_oauthlib/oauth2_session.py", line 547, in request
    token = self.refresh_token(
            ^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/requests_oauthlib/oauth2_session.py", line 496, in refresh_token
    self.token = self._client.parse_request_body_response(r.text, scope=self.scope)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 427, in parse_request_body_response
    self.token = parse_token_response(body, scope=scope)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/.venv/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 441, in parse_token_response
    validate_token_parameters(params)
  File "/opt/.venv/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 448, in validate_token_parameters
    raise_from_error(params.get('error'), params)
  File "/opt/.venv/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/errors.py", line 399, in raise_from_error
    raise cls(**kwargs)
oauthlib.oauth2.rfc6749.errors.InvalidClientIdError: (invalid_request) AADSTS900144: The request body must contain the following parameter: 'refresh_token'. Trace ID: xxx Correlation ID: xxx Timestamp: 2024-12-20 16:39:43Z

/opt/handler/core/email_handler.py is my own code, process_emails looks something like

while True:
    messages = account.inbox.all()
    if messages.count() > 0:
        # process messages
    sleep(10)

My main issue is what happens to the request, mainly in cases where the token expires when I'm trying to send an email, so I need to know if I have to retry or if the retry is happening somewhere in the library.

[ecederstrand]

Ah, it turns out someone stumbled upon this issue before: #1115

Can you check whether your versions of oauthlib and requests_oauthlib are up-to-date?

[evelinagkougklia]

requests_oauthlib is at 2.0.0 and oauthlib is at 3.2.2, both latest versions.