Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have a bug right now in our hotswap loras where if a users passes a given
extra_lora
for a prediction, and then removes thatextra_lora
but otherwise doesn't changelora_scale
orreplicate_weights
, theextra_lora
will persist.Any change to
lora_scale
,replicate_weights
, or adding a newextra_lora
should reset the weights (assuming that request maps to the instance that has theextra_lora
that's sticking around) - but still, this is the very definition of a weird, annoying bug.The fix below explicitly monitors for changes in extra_lora and handles it the same way that changes in lora_weights are handled. It also handles the case where
extra_lora_weights
are passed in andlora_weights
aren't; this is an edge case, but good to handle.