Support for Deepspeed-MII worker #2994
tungsontran
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Deepspeed-MII (https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen) claims that their inference throughput is much faster than vLLM. Will Factory dev team consider adopting it soon?
Beta Was this translation helpful? Give feedback.
All reactions