Mirrored_Repos/InvokeAI
1
0
Fork 0
mirror of https://github.com/invoke-ai/InvokeAI synced 2024-08-30 20:32:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add a callout about the hackiness of dropping tokens in the TextualInversionManager.

Browse Source
This commit is contained in:
Ryan Dick 2024-05-27 10:53:12 -04:00 committed by Kent Keirsey
parent 3aa1c8d3a8
commit 829b9ad66b
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
invokeai/backend/textual_inversion.py
View File

@ -109,6 +109,9 @@ class TextualInversionManager(BaseTextualInversionManager):
# compel.embeddings_provider.get_token_ids(), which first removes and then adds back the start and end tokens.
max_length = self.tokenizer.model_max_length - 2
if len(new_token_ids) > max_length:
# HACK: If TI token expansion causes us to exceed the max text encoder input length, we silently discard
# tokens. Token expansion should happen in a way that is compatible with compel's default handling of long
# prompts.
new_token_ids = new_token_ids[0:max_length]
return new_token_ids