Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Ansari LLM Generate a Non-Streamed Response #73

Open
OdyAsh opened this issue Nov 8, 2024 · 0 comments
Open

Make Ansari LLM Generate a Non-Streamed Response #73

OdyAsh opened this issue Nov 8, 2024 · 0 comments
Assignees
Labels
performance quality Issues related to the quality/polish/responsiveness of the app itself

Comments

@OdyAsh
Copy link
Collaborator

OdyAsh commented Nov 8, 2024

Currently, the ansari.py is coded in a way that always anticipates litellm to return a streamed response (e.g., many "yield" statements, etc.).

We want to refactor and update this code so that it incorporates non-streamed generation as well (which can be used with certain endpoints like WhatsApp, where a message has to be sent as a whole and can't be streamed, etc.).

Additionally, this seems to be affecting the final response time to WhatsApp users. See this issue for more info.

@OdyAsh OdyAsh converted this from a draft issue Nov 8, 2024
@OdyAsh OdyAsh moved this to Backlog in Ansari Work Nov 8, 2024
@OdyAsh OdyAsh self-assigned this Nov 12, 2024
@OdyAsh OdyAsh added the performance quality Issues related to the quality/polish/responsiveness of the app itself label Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance quality Issues related to the quality/polish/responsiveness of the app itself
Projects
Status: Backlog
Status: Todo
Development

No branches or pull requests

1 participant