Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request]: Add ability in Token Counter to retrieve Open AI cached_tokens #17372

Open
c64er4ever opened this issue Dec 26, 2024 · 0 comments · May be fixed by #17380
Open

[Feature Request]: Add ability in Token Counter to retrieve Open AI cached_tokens #17372

c64er4ever opened this issue Dec 26, 2024 · 0 comments · May be fixed by #17380
Labels
enhancement New feature or request triage Issue needs to be triaged/prioritized

Comments

@c64er4ever
Copy link

Feature Description

Open AI enables automatic Prompt Caching, as described here: https://platform.openai.com/docs/guides/prompt-caching
The number of cached tokens for a prompt is returned in the usage structure, in the "cached_tokens" field. For example:
"usage": {
"prompt_tokens": 2006,
"completion_tokens": 300,
"total_tokens": 2306,
"prompt_tokens_details": {
"cached_tokens": 1920
},
"completion_tokens_details": {
"reasoning_tokens": 0,
"accepted_prediction_tokens": 0,
"rejected_prediction_tokens": 0
}
}

The request is for Token Counter to enable to get this field in addition to prompt_tokens, completion_tokens and total_tokens.

Reason

I am not familiar with a way to access this field when using Open AI indirectly with Llamaindex.

Value of Feature

As mentioned in Open AI's docs, having access to this value enables to monitor metrics such as cache hit rates, latency, and the percentage of tokens cached to optimize prompt and caching strategy.

@c64er4ever c64er4ever added enhancement New feature or request triage Issue needs to be triaged/prioritized labels Dec 26, 2024
sangwonku pushed a commit to sangwonku/llama_index that referenced this issue Dec 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request triage Issue needs to be triaged/prioritized
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant