use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
var YJ = Object . defineProperty ; var KJ = ( e , t , n ) => t in e ? YJ ( e , t , { enumerable : ! 0 , configurable : ! 0 , writable : ! 0 , value : n } ) : e [ t ] = n ; var sn = ( e , t , n ) => ( KJ ( e , typeof t != "symbol" ? t + "" : t , n ) , n ) ; function _ze ( ) { import ( "data:text/javascript," ) } function qN ( e , t ) { for ( var n = 0 ; n < t . length ; n ++ ) { const r = t [ n ] ; if ( typeof r != "string" && ! Array . isArray ( r ) ) { for ( const i in r ) if ( i !== "default" && ! ( i in e ) ) { const o = Object . getOwnPropertyDescriptor ( r , i ) ; o && Object . defineProperty ( e , i , o . get ? o : { enumerable : ! 0 , get : ( ) => r [ i ] } ) } } } return Object . freeze ( Object . defineProperty ( e , Symbol . toStringTag , { value : "Module" } ) ) } ( function ( ) { const t = document . createElement ( "link" ) . relList ; if ( t && t . supports && t . supports ( "modulepreload" ) ) return ; for ( const i of document . querySelectorAll ( 'link[rel="modulepreload"]' ) ) r ( i ) ; new MutationObserver ( i => { for ( const o of i ) if ( o . type === "childList" ) for ( const a of o . addedNodes ) a . tagName === "LINK" && a . rel === "modulepreload" && r ( a ) } ) . observe ( document , { childList : ! 0 , subtree : ! 0 } ) ; function n ( i ) { const o = { } ; return i . integrity && ( o . integrity = i . integrity ) , i . referrerpolicy && ( o . referrerPolicy = i . referrerpolicy ) , i . crossorigin === "use-credentials" ? o . credentials = "include" : i . crossorigin === "anonymous" ? o . credentials = "omit" : o . credentials = "same-origin" , o } function r ( i ) { if ( i . ep ) return ; i . ep = ! 0 ; const o = n ( i ) ; fetch ( i . href , o ) } } ) ( ) ; var _o = typeof globalThis < "u" ? globalThis : typeof window < "u" ? window : typeof global < "u" ? global : typeof self < "u" ? self : { } ; function t _ ( e ) { return e && e . _ _esModule && Object . prototype . hasOwnProperty . call ( e , "default" ) ? e . default : e } var C = { exports : { } } , nn = { } ; / * *
2022-12-26 15:23:16 +00:00
* @ license React
* react . production . min . js
*
* Copyright ( c ) Facebook , Inc . and its affiliates .
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree .
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
* /var iy=Symbol.for("react.element"),XJ=Symbol.for("react.portal"),ZJ=Symbol.for("react.fragment"),QJ=Symbol.for("react.strict_mode"),JJ=Symbol.for("react.profiler"),eee=Symbol.for("react.provider"),tee=Symbol.for("react.context"),nee=Symbol.for("react.forward_ref"),ree=Symbol.for("react.suspense"),iee=Symbol.for("react.memo"),oee=Symbol.for("react.lazy"),ZT=Symbol.iterator;function aee(e){return e===null||typeof e!="object"?null:(e=ZT&&e[ZT]||e["@@iterator"],typeof e=="function"?e:null)}var YN={isMounted:function(){return!1},enqueueForceUpdate:function(){},enqueueReplaceState:function(){},enqueueSetState:function(){}},KN=Object.assign,XN={};function u1(e,t,n){this.props=e,this.context=t,this.refs=XN,this.updater=n||YN}u1.prototype.isReactComponent={};u1.prototype.setState=function(e,t){if(typeof e!="object"&&typeof e!="function"&&e!=null)throw Error("setState(...): takes an object of state variables to update or a function which returns an object of state variables.");this.updater.enqueueSetState(this,e,t,"setState")};u1.prototype.forceUpdate=function(e){this.updater.enqueueForceUpdate(this,e,"forceUpdate")};function ZN(){}ZN.prototype=u1.prototype;function n_(e,t,n){this.props=e,this.context=t,this.refs=XN,this.updater=n||YN}var r_=n_.prototype=new ZN;r_.constructor=n_;KN(r_,u1.prototype);r_.isPureReactComponent=!0;var QT=Array.isArray,QN=Object.prototype.hasOwnProperty,i_={current:null},JN={key:!0,ref:!0,__self:!0,__source:!0};function eB(e,t,n){var r,i={},o=null,a=null;if(t!=null)for(r in t.ref!==void 0&&(a=t.ref),t.key!==void 0&&(o=""+t.key),t)QN.call(t,r)&&!JN.hasOwnProperty(r)&&(i[r]=t[r]);var s=arguments.length-2;if(s===1)i.children=n;else if(1<s){for(var l=Array(s),u=0;u<s;u++)l[u]=arguments[u+2];i.children=l}if(e&&e.defaultProps)for(r in s=e.defaultProps,s)i[r]===void 0&&(i[r]=s[r]);return{$$typeof:iy,type:e,key:o,ref:a,props:i,_owner:i_.current}}function see(e,t){return{$$typeof:iy,type:e.type,key:t,ref:e.ref,props:e.props,_owner:e._owner}}function o_(e){return typeof e=="object"&&e!==null&&e.$$typeof===iy}function lee(e){var t={"=":"=0",":":"=2"};return"$"+e.replace(/ [ = : ] / g , function ( n ) { return t [ n ] } ) } var JT = /\/+/g ; function $w ( e , t ) { return typeof e == "object" && e !== null && e . key != null ? lee ( "" + e . key ) : t . toString ( 36 ) } function Wb ( e , t , n , r , i ) { var o = typeof e ; ( o === "undefined" || o === "boolean" ) && ( e = null ) ; var a = ! 1 ; if ( e === null ) a = ! 0 ; else switch ( o ) { case "string" : case "number" : a = ! 0 ; break ; case "object" : switch ( e . $$typeof ) { case iy : case XJ : a = ! 0 } } if ( a ) return a = e , i = i ( a ) , e = r === "" ? "." + $w ( a , 0 ) : r , QT ( i ) ? ( n = "" , e != null && ( n = e . replace ( JT , "$&/" ) + "/" ) , Wb ( i , t , n , "" , function ( u ) { return u } ) ) : i != null && ( o _ ( i ) && ( i = see ( i , n + ( ! i . key || a && a . key === i . key ? "" : ( "" + i . key ) . replace ( JT , "$&/" ) + "/" ) + e ) ) , t . push ( i ) ) , 1 ; if ( a = 0 , r = r === "" ? "." : r + ":" , QT ( e ) ) for ( var s = 0 ; s < e . length ; s ++ ) { o = e [ s ] ; var l = r + $w ( o , s ) ; a += Wb ( o , t , n , l , i ) } else if ( l = aee ( e ) , typeof l == "function" ) for ( e = l . call ( e ) , s = 0 ; ! ( o = e . next ( ) ) . done ; ) o = o . value , l = r + $w ( o , s ++ ) , a += Wb ( o , t , n , l , i ) ; else if ( o === "object" ) throw t = String ( e ) , Error ( "Objects are not valid as a React child (found: " + ( t === "[object Object]" ? "object with keys {" + Object . keys ( e ) . join ( ", " ) + "}" : t ) + "). If you meant to render a collection of children, use an array instead." ) ; return a } function T3 ( e , t , n ) { if ( e == null ) return e ; var r = [ ] , i = 0 ; return Wb ( e , r , "" , "" , function ( o ) { return t . call ( n , o , i ++ ) } ) , r } function uee ( e ) { if ( e . _status === - 1 ) { var t = e . _result ; t = t ( ) , t . then ( function ( n ) { ( e . _status === 0 || e . _status === - 1 ) && ( e . _status = 1 , e . _result = n ) } , function ( n ) { ( e . _status === 0 || e . _status === - 1 ) && ( e . _status = 2 , e . _result = n ) } ) , e . _status === - 1 && ( e . _status = 0 , e . _result = t ) } if ( e . _status === 1 ) return e . _result . default ; throw e . _result } var Oo = { current : null } , Gb = { transition : null } , cee = { ReactCurrentDispatcher : Oo , ReactCurrentBatchConfig : Gb , ReactCurrentOwner : i _ } ; nn . Children = { map : T3 , forEach : function ( e , t , n ) { T3 ( e , function ( ) { t . apply ( this , arguments ) } , n ) } , count : function ( e ) { var t = 0 ; return T3 ( e , function ( ) { t ++ } ) , t } , toArray : function ( e ) { return T3 ( e , function ( t ) { return t } ) || [ ] } , only : function ( e ) { if ( ! o _ ( e ) ) throw Error ( "React.Children.only expected to receive a single React element child." ) ; return e } } ; nn . Component = u1 ; nn . Fragment = ZJ ; nn .
2022-12-26 15:23:16 +00:00
* @ license React
* scheduler . production . min . js
*
* Copyright ( c ) Facebook , Inc . and its affiliates .
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree .
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
* / ( f u n c t i o n ( e ) { f u n c t i o n t ( U , X ) { v a r l e = U . l e n g t h ; U . p u s h ( X ) ; e : f o r ( ; 0 < l e ; ) { v a r f e = l e - 1 > > > 1 , S e = U [ f e ] ; i f ( 0 < i ( S e , X ) ) U [ f e ] = X , U [ l e ] = S e , l e = f e ; e l s e b r e a k e } } f u n c t i o n n ( U ) { r e t u r n U . l e n g t h = = = 0 ? n u l l : U [ 0 ] } f u n c t i o n r ( U ) { i f ( U . l e n g t h = = = 0 ) r e t u r n n u l l ; v a r X = U [ 0 ] , l e = U . p o p ( ) ; i f ( l e ! = = X ) { U [ 0 ] = l e ; e : f o r ( v a r f e = 0 , S e = U . l e n g t h , A e = S e > > > 1 ; f e < A e ; ) { v a r b e = 2 * ( f e + 1 ) - 1 , k e = U [ b e ] , D e = b e + 1 , J e = U [ D e ] ; i f ( 0 > i ( k e , l e ) ) D e < S e & & 0 > i ( J e , k e ) ? ( U [ f e ] = J e , U [ D e ] = l e , f e = D e ) : ( U [ f e ] = k e , U [ b e ] = l e , f e = b e ) ; e l s e i f ( D e < S e & & 0 > i ( J e , l e ) ) U [ f e ] = J e , U [ D e ] = l e , f e = D e ; e l s e b r e a k e } } r e t u r n X } f u n c t i o n i ( U , X ) { v a r l e = U . s o r t I n d e x - X . s o r t I n d e x ; r e t u r n l e ! = = 0 ? l e : U . i d - X . i d } i f ( t y p e o f p e r f o r m a n c e = = " o b j e c t " & & t y p e o f p e r f o r m a n c e . n o w = = " f u n c t i o n " ) { v a r o = p e r f o r m a n c e ; e . u n s t a b l e _ n o w = f u n c t i o n ( ) { r e t u r n o . n o w ( ) } } e l s e { v a r a = D a t e , s = a . n o w ( ) ; e . u n s t a b l e _ n o w = f u n c t i o n ( ) { r e t u r n a . n o w ( ) - s } } v a r l = [ ] , u = [ ] , f = 1 , p = n u l l , m = 3 , v = ! 1 , b = ! 1 , x = ! 1 , E = t y p e o f s e t T i m e o u t = = " f u n c t i o n " ? s e t T i m e o u t : n u l l , k = t y p e o f c l e a r T i m e o u t = = " f u n c t i o n " ? c l e a r T i m e o u t : n u l l , _ = t y p e o f s e t I m m e d i a t e < " u " ? s e t I m m e d i a t e : n u l l ; t y p e o f n a v i g a t o r < " u " & & n a v i g a t o r . s c h e d u l i n g ! = = v o i d 0 & & n a v i g a t o r . s c h e d u l i n g . i s I n p u t P e n d i n g ! = = v o i d 0 & & n a v i g a t o r . s c h e d u l i n g . i s I n p u t P e n d i n g . b i n d ( n a v i g a t o r . s c h e d u l i n g ) ; f u n c t i o n T ( U ) { f o r ( v a r X = n ( u ) ; X ! = = n u l l ; ) { i f ( X . c a l l b a c k = = = n u l l ) r ( u ) ; e l s e i f ( X . s t a r t T i m e < = U ) r ( u ) , X . s o r t I n d e x = X . e x p i r a t i o n T i m e , t ( l , X ) ; e l s e b r e a k ; X = n ( u ) } } f u n c t i o n A ( U ) { i f ( x = ! 1 , T ( U ) , ! b ) i f ( n ( l ) ! = = n u l l ) b = ! 0 , J ( R ) ; e l s e { v a r X = n ( u ) ; X ! = = n u l l & & Z ( A , X . s t a r t T i m e - U ) } } f u n c t i o n R ( U , X ) { b = ! 1 , x & & ( x = ! 1 , k ( B ) , B = - 1 ) , v = ! 0 ; v a r l e = m ; t r y { f o r ( T ( X ) , p = n ( l ) ; p ! = = n u l l & & ( ! ( p . e x p i r a t i o n T i m e > X ) | | U & & ! K ( ) ) ; ) { v a r f e = p . c a l l b a c k ; i f ( t y p e o f f e = = " f u n c t i o n " ) { p . c a l l b a c k = n u l l , m = p . p r i o r i t y L e v e l ; v a r S e = f e ( p . e x p i r a t i o n T i m e < = X ) ; X = e . u n s t a b l e _ n o w ( ) , t y p e o f S e = = " f u n c t i o n " ? p . c a l l b a c k = S e : p = = = n ( l ) & & r ( l ) , T ( X ) } e l s e r ( l ) ; p = n ( l ) } i f ( p ! = = n u l l ) v a r A e = ! 0 ; e l s e { v a r b e = n ( u ) ; b e ! = = n u l l & & Z ( A , b e . s t a r t T i m e - X ) , A e = ! 1 } r e t u r n A e } f i n a l l y { p = n u l l , m = l e , v = ! 1 } } v a r I = ! 1 , D = n u l l , B = - 1 , z = 5 , j = - 1 ; f u n c t i o n K ( ) { r e t u r n ! ( e . u n s t a b l e _ n o w ( ) - j < z ) } f u n c t i o n t e ( ) { i f ( D ! = = n u l l ) { v a r U = e . u n s t a b l e _ n o w ( ) ; j = U ; v a r X = ! 0 ; t r y { X = D ( ! 0 , U ) } f i n a l l y { X ? G ( ) : ( I = ! 1 , D = n u l l ) } } e l s e I = ! 1 } v a r G ; i f ( t y p e o f _ = = " f u n c t i o n " ) G = f u n c t i o n ( ) { _ ( t e ) } ; e l s e i f ( t y p e o f M e s s a g e C h a n n e l < " u " ) { v a r $ = n e w M e s s a g e C h a n n e l , V = $ . p o r t 2 ; $ . p o r t 1 . o n m e s s a g e = t e , G = f u n c t i o n ( ) { V . p o s t M e s s a g e ( n u l l ) } } e l s e G = f u n c t i o n ( ) { E ( t e , 0 ) } ; f u n c t i o n J ( U ) { D = U , I | | ( I = ! 0 , G ( ) ) } f u n c t i o n Z ( U , X ) { B = E ( f u n c t i o n ( ) { U ( e . u n s t a b l e _ n o w ( ) ) } , X ) } e . u n s t a b l e _ I d l e P r i o r i t y = 5 , e . u n s t a b l e _ I m m e d i a t e P r i o r i t y = 1 , e . u n s t a b l e _ L o w P r i o r i t y = 4 , e . u n s t a b l e _ N o r m a l P r i o r i t y = 3 , e . u n s t a b l e _ P r o f i l i n g = n u l l , e . u n s t a b l e _ U s e r B l o c k i n g P r i o r i t y = 2 , e . u n s t a b l e _ c a n c e l C a l l b a c k = f u n c t i o n ( U ) { U . c a l l b a c k = n u l l } , e . u n s t a b l e _ c o n t i n u e E x e c u t i o n = f u n c t i o n ( ) { b | | v | | ( b = ! 0 , J ( R ) ) } , e . u n s t a b l e _ f o r c e F r a m e R a t e = f u n c t i o n ( U ) { 0 > U | | 1 2 5 < U ? c o n s o l e . e r r o r ( " f o r c e F r a m e R a t e t a k e s a p o s i t i v e i n t b e t w e e n 0 a n d 1 2 5 , f o r c i n g f r a m e r a t e s h i g h e r t h a n 1 2 5 f p s i s n o t s u p p o r t e d " ) : z = 0 < U ? M a t h . f l o o r ( 1 e 3 / U ) : 5 } , e . u n s t a b l e _ g e t C u r r e n t P r i o r i t y L e v e l = f u n c t i o n ( ) { r e t u r n m } , e . u n s t a b l e _ g e t F i r s t C a l l b a c k N o d e = f u n c t i o n ( ) { r e t u r n n ( l ) } , e . u n s t a b l e _ n e x t = f u n c t i o n ( U ) { s w i t c h ( m ) { c a s e 1 : c a s e 2 : c a s e 3 : v a r X = 3 ; b r e a k ; d e f a u l t : X = m } v a r l e = m ; m = X ; t r y { r e t u r n U ( ) } f i n a l l y { m = l e } } , e . u n s t a b l e _ p a u s e E x e c u t i o n = f u n c t i o n ( ) { } , e . u n s t a b l e _ r e q u e s t P a i n t = f u n c t i o n ( ) { } , e . u n s t a b l e _ r u n W i t h P r i o r i t y = f u n c t i o n ( U , X ) { s w i t c h ( U ) { c a s e 1 : c a s e 2 : c a s e 3 : c a s e 4 : c a s e 5 : b r e a k ; d e f a u l t : U = 3 } v a r l e = m ; m = U ; t r y { r e t u r n X ( ) } f i n a l l y { m = l e } } , e . u n s t a b l e _ s c h e d u l e C a l l b a c k = f u n c t i o n ( U , X , l e ) { v a r f e = e . u n s t a b l e _ n o w ( ) ; s w i t c h ( t y p e o f l e = = " o b j e c t " & & l e ! = = n u l l ? ( l e = l e . d e l a y , l e = t y p e o f l e = = " n u m b e r " & & 0 < l e ? f e + l e : f e ) : l e = f e , U ) { c a s e 1 : v a r S e = - 1 ; b r e a k ; c a s e 2 : S e = 2 5 0 ; b r e a k ; c a s e 5 : S e = 1 0 7 3 7 4 1 8 2 3 ; b r e a k ; c a s e 4 : S e = 1 e 4 ; b r e a k ; d e f a u l t : S e = 5 e 3 } r e t u r n S e = l e + S e , U = { i d : f + + , c a l l b a c k : X , p r i o r i t y L e v e l : U , s t a r t T i m e : l e , e x p i r a t i o n T i m e : S e , s o r t I n d e x : - 1 } , l e > f e ? ( U . s o r t I n d e x = l e , t ( u , U ) , n ( l ) = = = n u l l & & U = = = n ( u ) & & ( x ? ( k ( B ) , B = - 1 ) : x = ! 0 , Z ( A , l e - f e ) ) ) : ( U . s o r t I n d e x = S e , t ( l , U ) , b | | v | | ( b = ! 0 , J ( R ) ) ) , U } , e . u n s t a b l e _ s h o u l d Y i e l d = K , e . u n s t a b l e _ w r a p C a l l b a c k = f u n c t i o n ( U ) { v a r X = m ; r e t u r n f u n c t i o n ( ) { v a r l e = m ; m = X ; t r y { r e t u r n U . a p p l y ( t h i s , a r g u m e n t s ) } f i n a l l y { m = l e } } } } ) ( t B ) ; ( f u n c t i o n ( e ) { e . e x p o r t s = t B } ) ( f g ) ; / * *
2022-12-26 15:23:16 +00:00
* @ license React
* react - dom . production . min . js
*
* Copyright ( c ) Facebook , Inc . and its affiliates .
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree .
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
* /var nB=C.exports,Ra=fg.exports;function ze(e){for(var t="https:/ / reactjs . org / docs / error - decoder . html ? invariant = "+e,n=1;n<arguments.length;n++)t+=" & args [ ] = "+encodeURIComponent(arguments[n]);return" Minified React error # "+e+" ; visit "+t+" for the full message or use the non - minified dev environment for full errors and additional helpful warnings . "}var rB=new Set,c2={};function op(e,t){zg(e,t),zg(e+" Capture ",t)}function zg(e,t){for(c2[e]=t,e=0;e<t.length;e++)rB.add(t[e])}var tc=!(typeof window>" u "||typeof window.document>" u "||typeof window.document.createElement>" u "),GC=Object.prototype.hasOwnProperty,dee=/^[:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD][:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD\-.0-9\u00B7\u0300-\u036F\u203F-\u2040]*$/,eL={},tL={};function fee(e){return GC.call(tL,e)?!0:GC.call(eL,e)?!1:dee.test(e)?tL[e]=!0:(eL[e]=!0,!1)}function hee(e,t,n,r){if(n!==null&&n.type===0)return!1;switch(typeof t){case" function ":case" symbol ":return!0;case" boolean ":return r?!1:n!==null?!n.acceptsBooleans:(e=e.toLowerCase().slice(0,5),e!==" data - "&&e!==" aria - ");default:return!1}}function pee(e,t,n,r){if(t===null||typeof t>" u "||hee(e,t,n,r))return!0;if(r)return!1;if(n!==null)switch(n.type){case 3:return!t;case 4:return t===!1;case 5:return isNaN(t);case 6:return isNaN(t)||1>t}return!1}function Mo(e,t,n,r,i,o,a){this.acceptsBooleans=t===2||t===3||t===4,this.attributeName=r,this.attributeNamespace=i,this.mustUseProperty=n,this.propertyName=e,this.type=t,this.sanitizeURL=o,this.removeEmptyString=a}var Ki={};" children dangerouslySetInnerHTML defaultValue defaultChecked innerHTML suppressContentEditableWarning suppressHydrationWarning style ".split(" ").forEach(function(e){Ki[e]=new Mo(e,0,!1,e,null,!1,!1)});[[" acceptCharset "," accept - charset "],[" className "," class "],[" htmlFor "," for "],[" httpEquiv "," http - equiv "]].forEach(function(e){var t=e[0];Ki[t]=new Mo(t,1,!1,e[1],null,!1,!1)});[" contentEditable "," draggable "," spellCheck "," value "].forEach(function(e){Ki[e]=new Mo(e,2,!1,e.toLowerCase(),null,!1,!1)});[" autoReverse "," externalResourcesRequired "," focusable "," preserveAlpha "].forEach(function(e){Ki[e]=new Mo(e,2,!1,e,null,!1,!1)});" allowFullScreen async autoFocus autoPlay controls default defer disabled disablePictureInPicture disableRemotePlayback formNoValidate hidden loop noModule noValidate open playsInline readOnly required reversed scoped seamless itemScope ".split(" ").forEach(function(e){Ki[e]=new Mo(e,3,!1,e.toLowerCase(),null,!1,!1)});[" checked "," multiple "," muted "," selected "].forEach(function(e){Ki[e]=new Mo(e,3,!0,e,null,!1,!1)});[" capture "," download "].forEach(function(e){Ki[e]=new Mo(e,4,!1,e,null,!1,!1)});[" cols "," rows "," size "," span "].forEach(function(e){Ki[e]=new Mo(e,6,!1,e,null,!1,!1)});[" rowSpan "," start "].forEach(function(e){Ki[e]=new Mo(e,5,!1,e.toLowerCase(),null,!1,!1)});var a_=/[\-:]([a-z])/g;function s_(e){return e[1].toUpperCase()}" accent - height alignment - baseline arabic - form baseline - shift cap - height clip - path clip - rule color - interpolation color - interpolation - filters color - profile color - rendering dominant - baseline enable - background fill - opacity fill - rule flood - color flood - opacity font - family font - size font - size - adjust font - stretch font - style font - variant font - weight glyph - name glyph - orientation - horizontal glyph - orientation - vertical horiz - adv - x horiz - origin - x image - rendering letter - spacing lighting - color marker - end marker - mid marker - start overline - position overline - thickness paint - order panose - 1 pointer - events rendering - intent shape - rendering stop - color stop - opacity strikethrough - position strikethrough - thickness stroke - dasharray stroke - dashoffset stroke - linecap stroke - linejoin stroke - miterlimit stroke - opacity stroke - width text - anchor text - decoration text - rendering underline - position underline - thickness unicode - bidi unicode - range units - per - em v - alphabetic v - hanging v - ideographic v - mathematical vector - effect
` +Fw+e}var zw=!1;function jw(e,t){if(!e||zw)return"";zw=!0;var n=Error.prepareStackTrace;Error.prepareStackTrace=void 0;try{if(t)if(t=function(){throw Error()},Object.defineProperty(t.prototype,"props",{set:function(){throw Error()}}),typeof Reflect=="object"&&Reflect.construct){try{Reflect.construct(t,[])}catch(u){var r=u}Reflect.construct(e,[],t)}else{try{t.call()}catch(u){r=u}e.call(t.prototype)}else{try{throw Error()}catch(u){r=u}e()}}catch(u){if(u&&r&&typeof u.stack=="string"){for(var i=u.stack.split( `
2022-12-26 15:23:16 +00:00
` ),o=r.stack.split( `
` ),a=i.length-1,s=o.length-1;1<=a&&0<=s&&i[a]!==o[s];)s--;for(;1<=a&&0<=s;a--,s--)if(i[a]!==o[s]){if(a!==1||s!==1)do if(a--,s--,0>s||i[a]!==o[s]){var l= `
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` +i[a].replace(" at new "," at ");return e.displayName&&l.includes("<anonymous>")&&(l=l.replace("<anonymous>",e.displayName)),l}while(1<=a&&0<=s);break}}}finally{zw=!1,Error.prepareStackTrace=n}return(e=e?e.displayName||e.name:"")?pv(e):""}function gee(e){switch(e.tag){case 5:return pv(e.type);case 16:return pv("Lazy");case 13:return pv("Suspense");case 19:return pv("SuspenseList");case 0:case 2:case 15:return e=jw(e.type,!1),e;case 11:return e=jw(e.type.render,!1),e;case 1:return e=jw(e.type,!0),e;default:return""}}function XC(e){if(e==null)return null;if(typeof e=="function")return e.displayName||e.name||null;if(typeof e=="string")return e;switch(e){case H0:return"Fragment";case j0:return"Portal";case qC:return"Profiler";case u_:return"StrictMode";case YC:return"Suspense";case KC:return"SuspenseList"}if(typeof e=="object")switch(e. $ $ typeof){case oB:return(e.displayName||"Context")+".Consumer";case iB:return(e._context.displayName||"Context")+".Provider";case c_:var t=e.render;return e=e.displayName,e||(e=t.displayName||t.name||"",e=e!==""?"ForwardRef("+e+")":"ForwardRef"),e;case d_:return t=e.displayName||null,t!==null?t:XC(e.type)||"Memo";case vd:t=e._payload,e=e._init;try{return XC(e(t))}catch(n){}}return null}function mee(e){var t=e.type;switch(e.tag){case 24:return"Cache";case 9:return(t.displayName||"Context")+".Consumer";case 10:return(t._context.displayName||"Context")+".Provider";case 18:return"DehydratedFragment";case 11:return e=t.render,e=e.displayName||e.name||"",t.displayName||(e!==""?"ForwardRef("+e+")":"ForwardRef");case 7:return"Fragment";case 5:return t;case 4:return"Portal";case 3:return"Root";case 6:return"Text";case 16:return XC(t);case 8:return t===u_?"StrictMode":"Mode";case 22:return"Offscreen";case 12:return"Profiler";case 21:return"Scope";case 13:return"Suspense";case 19:return"SuspenseList";case 25:return"TracingMarker";case 1:case 0:case 17:case 2:case 14:case 15:if(typeof t=="function")return t.displayName||t.name||null;if(typeof t=="string")return t}return null}function qd(e){switch(typeof e){case"boolean":case"number":case"string":case"undefined":return e;case"object":return e;default:return""}}function sB(e){var t=e.type;return(e=e.nodeName)&&e.toLowerCase()==="input"&&(t==="checkbox"||t==="radio")}function vee(e){var t=sB(e)?"checked":"value",n=Object.getOwnPropertyDescriptor(e.constructor.prototype,t),r=""+e[t];if(!e.hasOwnProperty(t)&&typeof n<"u"&&typeof n.get=="function"&&typeof n.set=="function"){var i=n.get,o=n.set;return Object.defineProperty(e,t,{configurable:!0,get:function(){return i.call(this)},set:function(a){r=""+a,o.call(this,a)}}),Object.defineProperty(e,t,{enumerable:n.enumerable}),{getValue:function(){return r},setValue:function(a){r=""+a},stopTracking:function(){e._valueTracker=null,delete e[t]}}}}function A3(e){e._valueTracker||(e._valueTracker=vee(e))}function lB(e){if(!e)return!1;var t=e._valueTracker;if(!t)return!0;var n=t.getValue(),r="";return e&&(r=sB(e)?e.checked?"true":"false":e.value),e=r,e!==n?(t.setValue(e),!0):!1}function I4(e){if(e=e||(typeof document<"u"?document:void 0),typeof e>"u")return null;try{return e.activeElement||e.body}catch(t){return e.body}}function ZC(e,t){var n=t.checked;return _r({},t,{defaultChecked:void 0,defaultValue:void 0,value:void 0,checked:n!=null?n:e._wrapperState.initialChecked})}function rL(e,t){var n=t.defaultValue==null?"":t.defaultValue,r=t.checked!=null?t.checked:t.defaultChecked;n=qd(t.value!=null?t.value:n),e._wrapperState={initialChecked:r,initialValue:n,controlled:t.type==="checkbox"||t.type==="radio"?t.checked!=null:t.value!=null}}function uB(e,t){t=t.checked,t!=null&&l_(e,"checked",t,!1)}function QC(e,t){uB(e,t);var n=qd(t.value),r=t.type;if(n!=null)r==="number"?(n===0&&e.value===""||e.value!=n)&&(e.value=""+n):e.value!==""+n&&(e.value=""+n);else if(r==="submit"||r==="reset"){e.removeAttribute("value");return}t.hasOwnProperty("value")?JC(e,t.type,n):t.hasOwnProperty("defaultValue")&&JC(e,t.type,qd(t.defaultValue)),t.checked==null&&t.defaultChecked!=null&&(e.defaultChecked=!!t.defaultChecked)}function iL(e,t,n){if(t.ha
` ).replace(Pte,"")}function $ 3(e,t,n){if(t=PL(t),PL(e)!==t&&n)throw Error(ze(425))}function H4(){}var h7=null,p7=null;function g7(e,t){return e==="textarea"||e==="noscript"||typeof t.children=="string"||typeof t.children=="number"||typeof t.dangerouslySetInnerHTML=="object"&&t.dangerouslySetInnerHTML!==null&&t.dangerouslySetInnerHTML.__html!=null}var m7=typeof setTimeout=="function"?setTimeout:void 0,Tte=typeof clearTimeout=="function"?clearTimeout:void 0,TL=typeof Promise=="function"?Promise:void 0,Lte=typeof queueMicrotask=="function"?queueMicrotask:typeof TL<"u"?function(e){return TL.resolve(null).then(e).catch(Ate)}:m7;function Ate(e){setTimeout(function(){throw e})}function e6(e,t){var n=t,r=0;do{var i=n.nextSibling;if(e.removeChild(n),i&&i.nodeType===8)if(n=i.data,n==="/ $ "){if(r===0){e.removeChild(i),g2(t);return}r--}else n!==" $ "&&n!==" $ ?"&&n!==" $ !"||r++;n=i}while(n);g2(t)}function Bd(e){for(;e!=null;e=e.nextSibling){var t=e.nodeType;if(t===1||t===3)break;if(t===8){if(t=e.data,t===" $ "||t===" $ !"||t===" $ ?")break;if(t==="/ $ ")return null}}return e}function LL(e){e=e.previousSibling;for(var t=0;e;){if(e.nodeType===8){var n=e.data;if(n===" $ "||n===" $ !"||n===" $ ?"){if(t===0)return e;t--}else n==="/ $ "&&t++}e=e.previousSibling}return null}var d1=Math.random().toString(36).slice(2),Vl="__reactFiber $ "+d1,S2="__reactProps $ "+d1,nc="__reactContainer $ "+d1,v7="__reactEvents $ "+d1,Ote="__reactListeners $ "+d1,Mte="__reactHandles $ "+d1;function Eh(e){var t=e[Vl];if(t)return t;for(var n=e.parentNode;n;){if(t=n[nc]||n[Vl]){if(n=t.alternate,t.child!==null||n!==null&&n.child!==null)for(e=LL(e);e!==null;){if(n=e[Vl])return n;e=LL(e)}return t}e=n,n=e.parentNode}return null}function sy(e){return e=e[Vl]||e[nc],!e||e.tag!==5&&e.tag!==6&&e.tag!==13&&e.tag!==3?null:e}function G0(e){if(e.tag===5||e.tag===6)return e.stateNode;throw Error(ze(33))}function cS(e){return e[S2]||null}var y7=[],q0=-1;function sf(e){return{current:e}}function sr(e){0>q0||(e.current=y7[q0],y7[q0]=null,q0--)}function tr(e,t){q0++,y7[q0]=e.current,e.current=t}var Yd={},uo=sf(Yd),Jo=sf(!1),Gh=Yd;function jg(e,t){var n=e.type.contextTypes;if(!n)return Yd;var r=e.stateNode;if(r&&r.__reactInternalMemoizedUnmaskedChildContext===t)return r.__reactInternalMemoizedMaskedChildContext;var i={},o;for(o in n)i[o]=t[o];return r&&(e=e.stateNode,e.__reactInternalMemoizedUnmaskedChildContext=t,e.__reactInternalMemoizedMaskedChildContext=i),i}function ea(e){return e=e.childContextTypes,e!=null}function V4(){sr(Jo),sr(uo)}function AL(e,t,n){if(uo.current!==Yd)throw Error(ze(168));tr(uo,t),tr(Jo,n)}function ZB(e,t,n){var r=e.stateNode;if(t=t.childContextTypes,typeof r.getChildContext!="function")return n;r=r.getChildContext();for(var i in r)if(!(i in t))throw Error(ze(108,mee(e)||"Unknown",i));return _r({},n,r)}function U4(e){return e=(e=e.stateNode)&&e.__reactInternalMemoizedMergedChildContext||Yd,Gh=uo.current,tr(uo,e),tr(Jo,Jo.current),!0}function OL(e,t,n){var r=e.stateNode;if(!r)throw Error(ze(169));n?(e=ZB(e,t,Gh),r.__reactInternalMemoizedMergedChildContext=e,sr(Jo),sr(uo),tr(uo,e)):sr(Jo),tr(Jo,n)}var Gu=null,dS=!1,t6=!1;function QB(e){Gu===null?Gu=[e]:Gu.push(e)}function Rte(e){dS=!0,QB(e)}function lf(){if(!t6&&Gu!==null){t6=!0;var e=0,t=In;try{var n=Gu;for(In=1;e<n.length;e++){var r=n[e];do r=r(!0);while(r!==null)}Gu=null,dS=!1}catch(i){throw Gu!==null&&(Gu=Gu.slice(e+1)),CB(h_,lf),i}finally{In=t,t6=!1}}return null}var Y0=[],K0=0,W4=null,G4=0,es=[],ts=0,qh=null,Yu=1,Ku="";function gh(e,t){Y0[K0++]=G4,Y0[K0++]=W4,W4=e,G4=t}function JB(e,t,n){es[ts++]=Yu,es[ts++]=Ku,es[ts++]=qh,qh=e;var r=Yu;e=Ku;var i=32-Ys(r)-1;r&=~(1<<i),n+=1;var o=32-Ys(t)+i;if(30<o){var a=i-i%5;o=(r&(1<<a)-1).toString(32),r>>=a,i-=a,Yu=1<<32-Ys(t)+i|n<<i|r,Ku=o+e}else Yu=1<<o|n<<i|r,Ku=e}function w_(e){e.return!==null&&(gh(e,1),JB(e,1,0))}function C_(e){for(;e===W4;)W4=Y0[--K0],Y0[K0]=null,G4=Y0[--K0],Y0[K0]=null;for(;e===qh;)qh=es[--ts],es[ts]=null,Ku=es[--ts],es[ts]=null,Yu=es[--ts],es[ts]=null}var La=null,Ta=null,fr=!1,Us=null;function e $ (e,t){var n=os(5,null,null,0);n.elementType="DELETED",n.stateNode=t,n.return
2022-12-26 15:23:16 +00:00
Error generating stack : ` +o.message+ `
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` +o.stack}return{value:e,source:t,stack:i,digest:null}}function a6(e,t,n){return{value:e,source:null,stack:n!=null?n:null,digest:t!=null?t:null}}function _7(e,t){try{console.error(t.value)}catch(n){setTimeout(function(){throw n})}}var Hte=typeof WeakMap=="function"?WeakMap:Map;function T $ (e,t,n){n=Zu(-1,n),n.tag=3,n.payload={element:null};var r=t.value;return n.callback=function(){e5||(e5=!0,I7=r),_7(e,t)},n}function L $ (e,t,n){n=Zu(-1,n),n.tag=3;var r=e.type.getDerivedStateFromError;if(typeof r=="function"){var i=t.value;n.payload=function(){return r(i)},n.callback=function(){_7(e,t)}}var o=e.stateNode;return o!==null&&typeof o.componentDidCatch=="function"&&(n.callback=function(){_7(e,t),typeof r!="function"&&(Fd===null?Fd=new Set([this]):Fd.add(this));var a=t.stack;this.componentDidCatch(t.value,{componentStack:a!==null?a:""})}),n}function jL(e,t,n){var r=e.pingCache;if(r===null){r=e.pingCache=new Hte;var i=new Set;r.set(t,i)}else i=r.get(t),i===void 0&&(i=new Set,r.set(t,i));i.has(n)||(i.add(n),e=nne.bind(null,e,t,n),t.then(e,e))}function HL(e){do{var t;if((t=e.tag===13)&&(t=e.memoizedState,t=t!==null?t.dehydrated!==null:!0),t)return e;e=e.return}while(e!==null);return null}function VL(e,t,n,r,i){return(e.mode&1)===0?(e===t?e.flags|=65536:(e.flags|=128,n.flags|=131072,n.flags&=-52805,n.tag===1&&(n.alternate===null?n.tag=17:(t=Zu(-1,1),t.tag=2, $ d(n,t,1))),n.lanes|=1),e):(e.flags|=65536,e.lanes=i,e)}var Vte=pc.ReactCurrentOwner,Zo=!1;function Co(e,t,n,r){t.child=e===null?s $ (t,null,n,r):Vg(t,e.child,n,r)}function UL(e,t,n,r,i){n=n.render;var o=t.ref;return vg(t,i),r=I_(e,t,n,r,o,i),n=D_(),e!==null&&!Zo?(t.updateQueue=e.updateQueue,t.flags&=-2053,e.lanes&=~i,ic(e,t,i)):(fr&&n&&w_(t),t.flags|=1,Co(e,t,r,i),t.child)}function WL(e,t,n,r,i){if(e===null){var o=n.type;return typeof o=="function"&&!W_(o)&&o.defaultProps===void 0&&n.compare===null&&n.defaultProps===void 0?(t.tag=15,t.type=o,A $ (e,t,o,r,i)):(e=n4(n.type,null,r,t,t.mode,i),e.ref=t.ref,e.return=t,t.child=e)}if(o=e.child,(e.lanes&i)===0){var a=o.memoizedProps;if(n=n.compare,n=n!==null?n:v2,n(a,r)&&e.ref===t.ref)return ic(e,t,i)}return t.flags|=1,e=jd(o,r),e.ref=t.ref,e.return=t,t.child=e}function A $ (e,t,n,r,i){if(e!==null){var o=e.memoizedProps;if(v2(o,r)&&e.ref===t.ref)if(Zo=!1,t.pendingProps=r=o,(e.lanes&i)!==0)(e.flags&131072)!==0&&(Zo=!0);else return t.lanes=e.lanes,ic(e,t,i)}return k7(e,t,n,r,i)}function O $ (e,t,n){var r=t.pendingProps,i=r.children,o=e!==null?e.memoizedState:null;if(r.mode==="hidden")if((t.mode&1)===0)t.memoizedState={baseLanes:0,cachePool:null,transitions:null},tr(Q0,Ea),Ea|=n;else{if((n&1073741824)===0)return e=o!==null?o.baseLanes|n:n,t.lanes=t.childLanes=1073741824,t.memoizedState={baseLanes:e,cachePool:null,transitions:null},t.updateQueue=null,tr(Q0,Ea),Ea|=e,null;t.memoizedState={baseLanes:0,cachePool:null,transitions:null},r=o!==null?o.baseLanes:n,tr(Q0,Ea),Ea|=r}else o!==null?(r=o.baseLanes|n,t.memoizedState=null):r=n,tr(Q0,Ea),Ea|=r;return Co(e,t,i,n),t.child}function M $ (e,t){var n=t.ref;(e===null&&n!==null||e!==null&&e.ref!==n)&&(t.flags|=512,t.flags|=2097152)}function k7(e,t,n,r,i){var o=ea(n)?Gh:uo.current;return o=jg(t,o),vg(t,i),n=I_(e,t,n,r,o,i),r=D_(),e!==null&&!Zo?(t.updateQueue=e.updateQueue,t.flags&=-2053,e.lanes&=~i,ic(e,t,i)):(fr&&r&&w_(t),t.flags|=1,Co(e,t,n,i),t.child)}function GL(e,t,n,r,i){if(ea(n)){var o=!0;U4(t)}else o=!1;if(vg(t,i),t.stateNode===null)Jb(e,t),o $ (t,n,r),C7(t,n,r,i),r=!0;else if(e===null){var a=t.stateNode,s=t.memoizedProps;a.props=s;var l=a.context,u=n.contextType;typeof u=="object"&&u!==null?u=cs(u):(u=ea(n)?Gh:uo.current,u=jg(t,u));var f=n.getDerivedStateFromProps,p=typeof f=="function"||typeof a.getSnapshotBeforeUpdate=="function";p||typeof a.UNSAFE_componentWillReceiveProps!="function"&&typeof a.componentWillReceiveProps!="function"||(s!==r||l!==u)&&BL(t,a,r,u),yd=!1;var m=t.memoizedState;a.state=m,K4(t,r,a,i),l=t.memoizedState,s!==r||m!==l||Jo.current||yd?(typeof f=="function"&&(w7(t,n,f,r),l=t.memoizedState),(s=yd||NL(t,n,s,r,m,l,u))?(p||typeof a.UNSAFE_componentWillMount!="function"&&typeof a.comp
2022-12-26 15:23:16 +00:00
* @ license React
* react - jsx - runtime . production . min . js
*
* Copyright ( c ) Facebook , Inc . and its affiliates .
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree .
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
* / v a r h n e = C . e x p o r t s , p n e = S y m b o l . f o r ( " r e a c t . e l e m e n t " ) , g n e = S y m b o l . f o r ( " r e a c t . f r a g m e n t " ) , m n e = O b j e c t . p r o t o t y p e . h a s O w n P r o p e r t y , v n e = h n e . _ _ S E C R E T _ I N T E R N A L S _ D O _ N O T _ U S E _ O R _ Y O U _ W I L L _ B E _ F I R E D . R e a c t C u r r e n t O w n e r , y n e = { k e y : ! 0 , r e f : ! 0 , _ _ s e l f : ! 0 , _ _ s o u r c e : ! 0 } ; f u n c t i o n t F ( e , t , n ) { v a r r , i = { } , o = n u l l , a = n u l l ; n ! = = v o i d 0 & & ( o = " " + n ) , t . k e y ! = = v o i d 0 & & ( o = " " + t . k e y ) , t . r e f ! = = v o i d 0 & & ( a = t . r e f ) ; f o r ( r i n t ) m n e . c a l l ( t , r ) & & ! y n e . h a s O w n P r o p e r t y ( r ) & & ( i [ r ] = t [ r ] ) ; i f ( e & & e . d e f a u l t P r o p s ) f o r ( r i n t = e . d e f a u l t P r o p s , t ) i [ r ] = = = v o i d 0 & & ( i [ r ] = t [ r ] ) ; r e t u r n { $ $ t y p e o f : p n e , t y p e : e , k e y : o , r e f : a , p r o p s : i , _ o w n e r : v n e . c u r r e n t } } w S . F r a g m e n t = g n e ; w S . j s x = t F ; w S . j s x s = t F ; ( f u n c t i o n ( e ) { e . e x p o r t s = w S } ) ( x S ) ; c o n s t f n = x S . e x p o r t s . F r a g m e n t , y = x S . e x p o r t s . j s x , q = x S . e x p o r t s . j s x s ; v a r X _ = C . e x p o r t s . c r e a t e C o n t e x t ( { } ) ; X _ . d i s p l a y N a m e = " C o l o r M o d e C o n t e x t " ; f u n c t i o n u y ( ) { c o n s t e = C . e x p o r t s . u s e C o n t e x t ( X _ ) ; i f ( e = = = v o i d 0 ) t h r o w n e w E r r o r ( " u s e C o l o r M o d e m u s t b e u s e d w i t h i n a C o l o r M o d e P r o v i d e r " ) ; r e t u r n e } v a r G 3 = { l i g h t : " c h a k r a - u i - l i g h t " , d a r k : " c h a k r a - u i - d a r k " } ; f u n c t i o n b n e ( e = { } ) { c o n s t { p r e v e n t T r a n s i t i o n : t = ! 0 } = e , n = { s e t D a t a s e t : r = > { c o n s t i = t ? n . p r e v e n t T r a n s i t i o n ( ) : v o i d 0 ; d o c u m e n t . d o c u m e n t E l e m e n t . d a t a s e t . t h e m e = r , d o c u m e n t . d o c u m e n t E l e m e n t . s t y l e . c o l o r S c h e m e = r , i = = n u l l | | i ( ) } , s e t C l a s s N a m e ( r ) { d o c u m e n t . b o d y . c l a s s L i s t . a d d ( r ? G 3 . d a r k : G 3 . l i g h t ) , d o c u m e n t . b o d y . c l a s s L i s t . r e m o v e ( r ? G 3 . l i g h t : G 3 . d a r k ) } , q u e r y ( ) { r e t u r n w i n d o w . m a t c h M e d i a ( " ( p r e f e r s - c o l o r - s c h e m e : d a r k ) " ) } , g e t S y s t e m T h e m e ( r ) { v a r o ; r e t u r n ( ( o = n . q u e r y ( ) . m a t c h e s ) ! = n u l l ? o : r = = = " d a r k " ) ? " d a r k " : " l i g h t " } , a d d L i s t e n e r ( r ) { c o n s t i = n . q u e r y ( ) , o = a = > { r ( a . m a t c h e s ? " d a r k " : " l i g h t " ) } ; r e t u r n t y p e o f i . a d d L i s t e n e r = = " f u n c t i o n " ? i . a d d L i s t e n e r ( o ) : i . a d d E v e n t L i s t e n e r ( " c h a n g e " , o ) , ( ) = > { t y p e o f i . r e m o v e L i s t e n e r = = " f u n c t i o n " ? i . r e m o v e L i s t e n e r ( o ) : i . r e m o v e E v e n t L i s t e n e r ( " c h a n g e " , o ) } } , p r e v e n t T r a n s i t i o n ( ) { c o n s t r = d o c u m e n t . c r e a t e E l e m e n t ( " s t y l e " ) ; r e t u r n r . a p p e n d C h i l d ( d o c u m e n t . c r e a t e T e x t N o d e ( " * { - w e b k i t - t r a n s i t i o n : n o n e ! i m p o r t a n t ; - m o z - t r a n s i t i o n : n o n e ! i m p o r t a n t ; - o - t r a n s i t i o n : n o n e ! i m p o r t a n t ; - m s - t r a n s i t i o n : n o n e ! i m p o r t a n t ; t r a n s i t i o n : n o n e ! i m p o r t a n t } " ) ) , d o c u m e n t . h e a d . a p p e n d C h i l d ( r ) , ( ) = > { w i n d o w . g e t C o m p u t e d S t y l e ( d o c u m e n t . b o d y ) , r e q u e s t A n i m a t i o n F r a m e ( ( ) = > { r e q u e s t A n i m a t i o n F r a m e ( ( ) = > { d o c u m e n t . h e a d . r e m o v e C h i l d ( r ) } ) } ) } } } ; r e t u r n n } v a r S n e = " c h a k r a - u i - c o l o r - m o d e " ; f u n c t i o n x n e ( e ) { r e t u r n { s s r : ! 1 , t y p e : " l o c a l S t o r a g e " , g e t ( t ) { i f ( ! ( g l o b a l T h i s ! = n u l l & & g l o b a l T h i s . d o c u m e n t ) ) r e t u r n t ; l e t n ; t r y { n = l o c a l S t o r a g e . g e t I t e m ( e ) | | t } c a t c h ( r ) { } r e t u r n n | | t } , s e t ( t ) { t r y { l o c a l S t o r a g e . s e t I t e m ( e , t ) } c a t c h ( n ) { } } } } v a r w n e = x n e ( S n e ) , a A = ( ) = > { } ; f u n c t i o n s A ( e , t ) { r e t u r n e . t y p e = = = " c o o k i e " & & e . s s r ? e . g e t ( t ) : t } f u n c t i o n n F ( e ) { c o n s t { v a l u e : t , c h i l d r e n : n , o p t i o n s : { u s e S y s t e m C o l o r M o d e : r , i n i t i a l C o l o r M o d e : i , d i s a b l e T r a n s i t i o n O n C h a n g e : o } = { } , c o l o r M o d e M a n a g e r : a = w n e } = e , s = i = = = " d a r k " ? " d a r k " : " l i g h t " , [ l , u ] = C . e x p o r t s . u s e S t a t e ( ( ) = > s A ( a , s ) ) , [ f , p ] = C . e x p o r t s . u s e S t a t e ( ( ) = > s A ( a ) ) , { g e t S y s t e m T h e m e : m , s e t C l a s s N a m e : v , s e t D a t a s e t : b , a d d L i s t e n e r : x } = C . e x p o r t s . u s e M e m o ( ( ) = > b n e ( { p r e v e n t T r a n s i t i o n : o } ) , [ o ] ) , E = i = = = " s y s t e m " & & ! l ? f : l , k = C . e x p o r t s . u s e C a l l b a c k ( A = > { c o n s t R = A = = = " s y s t e m " ? m ( ) : A ; u ( R ) , v ( R = = = " d a r k " ) , b ( R ) , a . s e t ( R ) } , [ a , m , v , b ] ) ; X s ( ( ) = > { i = = = " s y s t e m " & & p ( m ( ) ) } , [ ] ) , C . e x p o r t s . u s e E f f e c t ( ( ) = > { c o n s t A = a . g e t ( ) ; i f ( A ) { k ( A ) ; r e t u r n } i f ( i = = = " s y s t e m " ) { k ( " s y s t e m " ) ; r e t u r n } k ( s ) } , [ a , s , i , k ] ) ; c o n s t _ = C . e x p o r t s . u s e C a l l b a c k ( ( ) = > { k ( E = = = " d a r k " ? " l i g h t " : " d a r k " ) } , [ E , k ] ) ; C . e x p o r t s . u s e E f f e c t ( ( ) = > { i f ( ! ! r ) r e t u r n x ( k ) } , [ r , x , k ] ) ; c o n s t T = C . e x p o r t s . u s e M e m o ( ( ) = > ( { c o l o r M o d e : t ! = n u l l ? t : E , t o g g l e C o l o r M o d e : t ? a A : _ , s e t C o l o r M o d e : t ? a A : k , f o r c e d : t ! = = v o i d 0 } ) , [ E , _ , k , t ] ) ; r e t u r n y ( X _ . P r o v i d e r , { v a l u e : T , c h i l d r e n : n } ) } n F . d i s p l a y N a m e = " C o l o r M o d e P r o v i d e r " ; v a r $ 7 = { e x p o r t s : { } } ; ( f u n c t i o n ( e , t ) { v a r n = 2 0 0 , r = " _ _ l o d a s h _ h a s h _ u n d e f i n e d _ _ " , i = 8 0 0 , o = 1 6 , a = 9 0 0 7 1 9 9 2 5 4 7 4 0 9 9 1 , s = " [ o b j e c t A r g u m e n t s ] " , l = " [ o b j e c t A r r a y ] " , u = " [ o b j e c t A s y n c F u n c t i o n ] " , f = " [ o b j e c t B o o l e a n ] " , p = " [ o b j e c t D a t e ] " , m = " [ o b j e c t E r r o r ] " , v = " [ o b j e c t F u n c t i o n ] " , b = " [ o b j e c t G e n e r a t o r F u n c t i o n ] " , x = " [ o b j e c t M a p ] " , E = " [ o b j e c t N u m b e r ] " , k = " [ o b j e c t N u l l ] " , _ = " [ o b j e c t O b j e c t ] " , T = " [ o b j e c t P r o x y ] " , A = " [ o b j e c t R e g E x p ] " , R = " [ o b j e c t S e t ] " , I = " [ o b j e c t S t r i n g ] " , D = " [ o b j e c t U n d e f i n e d ] " , B = " [ o b j e c t W e a k M a p ] " , z = " [ o b j e c t A r r a y B u f f e r ] " , j = " [ o b j e c t D a t a V i e w ] " , K = " [ o b j e c t F l o a t 3 2 A r r a y ] " , t e = " [ o b j e c t F l o a t 6 4 A r r a y ] " , G = " [ o b j e c t I n t 8 A r r a y ] " , $ = " [ o b j e c t I n t 1 6 A r r a y ] " , V = " [ o b j e c t I n t 3 2 A r r a y ] " , J = " [ o b j e
2022-12-26 15:23:16 +00:00
* react - is . production . min . js
*
* Copyright ( c ) Facebook , Inc . and its affiliates .
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree .
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
* / v a r A i = t y p e o f S y m b o l = = " f u n c t i o n " & & S y m b o l . f o r , n k = A i ? S y m b o l . f o r ( " r e a c t . e l e m e n t " ) : 6 0 1 0 3 , r k = A i ? S y m b o l . f o r ( " r e a c t . p o r t a l " ) : 6 0 1 0 6 , T S = A i ? S y m b o l . f o r ( " r e a c t . f r a g m e n t " ) : 6 0 1 0 7 , L S = A i ? S y m b o l . f o r ( " r e a c t . s t r i c t _ m o d e " ) : 6 0 1 0 8 , A S = A i ? S y m b o l . f o r ( " r e a c t . p r o f i l e r " ) : 6 0 1 1 4 , O S = A i ? S y m b o l . f o r ( " r e a c t . p r o v i d e r " ) : 6 0 1 0 9 , M S = A i ? S y m b o l . f o r ( " r e a c t . c o n t e x t " ) : 6 0 1 1 0 , i k = A i ? S y m b o l . f o r ( " r e a c t . a s y n c _ m o d e " ) : 6 0 1 1 1 , R S = A i ? S y m b o l . f o r ( " r e a c t . c o n c u r r e n t _ m o d e " ) : 6 0 1 1 1 , I S = A i ? S y m b o l . f o r ( " r e a c t . f o r w a r d _ r e f " ) : 6 0 1 1 2 , D S = A i ? S y m b o l . f o r ( " r e a c t . s u s p e n s e " ) : 6 0 1 1 3 , h i e = A i ? S y m b o l . f o r ( " r e a c t . s u s p e n s e _ l i s t " ) : 6 0 1 2 0 , N S = A i ? S y m b o l . f o r ( " r e a c t . m e m o " ) : 6 0 1 1 5 , B S = A i ? S y m b o l . f o r ( " r e a c t . l a z y " ) : 6 0 1 1 6 , p i e = A i ? S y m b o l . f o r ( " r e a c t . b l o c k " ) : 6 0 1 2 1 , g i e = A i ? S y m b o l . f o r ( " r e a c t . f u n d a m e n t a l " ) : 6 0 1 1 7 , m i e = A i ? S y m b o l . f o r ( " r e a c t . r e s p o n d e r " ) : 6 0 1 1 8 , v i e = A i ? S y m b o l . f o r ( " r e a c t . s c o p e " ) : 6 0 1 1 9 ; f u n c t i o n F a ( e ) { i f ( t y p e o f e = = " o b j e c t " & & e ! = = n u l l ) { v a r t = e . $ $ t y p e o f ; s w i t c h ( t ) { c a s e n k : s w i t c h ( e = e . t y p e , e ) { c a s e i k : c a s e R S : c a s e T S : c a s e A S : c a s e L S : c a s e D S : r e t u r n e ; d e f a u l t : s w i t c h ( e = e & & e . $ $ t y p e o f , e ) { c a s e M S : c a s e I S : c a s e B S : c a s e N S : c a s e O S : r e t u r n e ; d e f a u l t : r e t u r n t } } c a s e r k : r e t u r n t } } } f u n c t i o n w F ( e ) { r e t u r n F a ( e ) = = = R S } N n . A s y n c M o d e = i k ; N n . C o n c u r r e n t M o d e = R S ; N n . C o n t e x t C o n s u m e r = M S ; N n . C o n t e x t P r o v i d e r = O S ; N n . E l e m e n t = n k ; N n . F o r w a r d R e f = I S ; N n . F r a g m e n t = T S ; N n . L a z y = B S ; N n . M e m o = N S ; N n . P o r t a l = r k ; N n . P r o f i l e r = A S ; N n . S t r i c t M o d e = L S ; N n . S u s p e n s e = D S ; N n . i s A s y n c M o d e = f u n c t i o n ( e ) { r e t u r n w F ( e ) | | F a ( e ) = = = i k } ; N n . i s C o n c u r r e n t M o d e = w F ; N n . i s C o n t e x t C o n s u m e r = f u n c t i o n ( e ) { r e t u r n F a ( e ) = = = M S } ; N n . i s C o n t e x t P r o v i d e r = f u n c t i o n ( e ) { r e t u r n F a ( e ) = = = O S } ; N n . i s E l e m e n t = f u n c t i o n ( e ) { r e t u r n t y p e o f e = = " o b j e c t " & & e ! = = n u l l & & e . $ $ t y p e o f = = = n k } ; N n . i s F o r w a r d R e f = f u n c t i o n ( e ) { r e t u r n F a ( e ) = = = I S } ; N n . i s F r a g m e n t = f u n c t i o n ( e ) { r e t u r n F a ( e ) = = = T S } ; N n . i s L a z y = f u n c t i o n ( e ) { r e t u r n F a ( e ) = = = B S } ; N n . i s M e m o = f u n c t i o n ( e ) { r e t u r n F a ( e ) = = = N S } ; N n . i s P o r t a l = f u n c t i o n ( e ) { r e t u r n F a ( e ) = = = r k } ; N n . i s P r o f i l e r = f u n c t i o n ( e ) { r e t u r n F a ( e ) = = = A S } ; N n . i s S t r i c t M o d e = f u n c t i o n ( e ) { r e t u r n F a ( e ) = = = L S } ; N n . i s S u s p e n s e = f u n c t i o n ( e ) { r e t u r n F a ( e ) = = = D S } ; N n . i s V a l i d E l e m e n t T y p e = f u n c t i o n ( e ) { r e t u r n t y p e o f e = = " s t r i n g " | | t y p e o f e = = " f u n c t i o n " | | e = = = T S | | e = = = R S | | e = = = A S | | e = = = L S | | e = = = D S | | e = = = h i e | | t y p e o f e = = " o b j e c t " & & e ! = = n u l l & & ( e . $ $ t y p e o f = = = B S | | e . $ $ t y p e o f = = = N S | | e . $ $ t y p e o f = = = O S | | e . $ $ t y p e o f = = = M S | | e . $ $ t y p e o f = = = I S | | e . $ $ t y p e o f = = = g i e | | e . $ $ t y p e o f = = = m i e | | e . $ $ t y p e o f = = = v i e | | e . $ $ t y p e o f = = = p i e ) } ; N n . t y p e O f = F a ; ( f u n c t i o n ( e ) { e . e x p o r t s = N n } ) ( x F ) ; v a r C F = x F . e x p o r t s , y i e = { $ $ t y p e o f : ! 0 , r e n d e r : ! 0 , d e f a u l t P r o p s : ! 0 , d i s p l a y N a m e : ! 0 , p r o p T y p e s : ! 0 } , b i e = { $ $ t y p e o f : ! 0 , c o m p a r e : ! 0 , d e f a u l t P r o p s : ! 0 , d i s p l a y N a m e : ! 0 , p r o p T y p e s : ! 0 , t y p e : ! 0 } , _ F = { } ; _ F [ C F . F o r w a r d R e f ] = y i e ; _ F [ C F . M e m o ] = b i e ; v a r S i e = ! 0 ; f u n c t i o n x i e ( e , t , n ) { v a r r = " " ; r e t u r n n . s p l i t ( " " ) . f o r E a c h ( f u n c t i o n ( i ) { e [ i ] ! = = v o i d 0 ? t . p u s h ( e [ i ] + " ; " ) : r + = i + " " } ) , r } v a r k F = f u n c t i o n ( t , n , r ) { v a r i = t . k e y + " - " + n . n a m e ; ( r = = = ! 1 | | S i e = = = ! 1 ) & & t . r e g i s t e r e d [ i ] = = = v o i d 0 & & ( t . r e g i s t e r e d [ i ] = n . s t y l e s ) } , E F = f u n c t i o n ( t , n , r ) { k F ( t , n , r ) ; v a r i = t . k e y + " - " + n . n a m e ; i f ( t . i n s e r t e d [ n . n a m e ] = = = v o i d 0 ) { v a r o = n ; d o t . i n s e r t ( n = = = o ? " . " + i : " " , o , t . s h e e t , ! 0 ) , o = o . n e x t ; w h i l e ( o ! = = v o i d 0 ) } } ; f u n c t i o n w i e ( e ) { f o r ( v a r t = 0 , n , r = 0 , i = e . l e n g t h ; i > = 4 ; + + r , i - = 4 ) n = e . c h a r C o d e A t ( r ) & 2 5 5 | ( e . c h a r C o d e A t ( + + r ) & 2 5 5 ) < < 8 | ( e . c h a r C o d e A t ( + + r ) & 2 5 5 ) < < 1 6 | ( e . c h a r C o d e A t ( + + r ) & 2 5 5 ) < < 2 4 , n = ( n & 6 5 5 3 5 ) * 1 5 4 0 4 8 3 4 7 7 + ( ( n > > > 1 6 ) * 5 9 7 9 7 < < 1 6 ) , n ^ = n > > > 2 4 , t = ( n & 6 5 5 3 5 ) * 1 5 4 0 4 8 3 4 7 7 + ( ( n > > > 1 6 ) * 5 9 7 9 7 < < 1 6 ) ^ ( t & 6 5 5 3 5 ) * 1 5 4 0 4 8 3 4 7 7 + ( ( t > > > 1 6 ) * 5 9 7 9 7 < < 1 6 ) ; s w i t c h ( i ) { c a s e 3 : t ^ = ( e . c h a r C o d e A t ( r + 2 ) & 2 5 5 ) < < 1 6 ; c a s e 2 : t ^ = ( e . c h a r C o d e A t ( r + 1 ) & 2 5 5 ) < < 8 ; c a s e 1 : t ^ = e . c h a r C o d e A t ( r ) & 2 5 5 , t = ( t & 6 5 5 3 5 ) * 1 5 4 0 4 8 3 4 7 7 + ( ( t > > > 1 6 ) * 5 9 7 9 7 < < 1 6 ) } r e t u r n t ^ = t > > > 1 3 , t = ( t & 6 5 5 3 5 ) * 1 5 4 0 4 8 3 4 7 7 + ( ( t > > > 1 6 ) * 5 9 7 9 7 < < 1 6 ) , ( ( t ^ t > > > 1 5 ) > > > 0 ) . t o S t r i n g ( 3 6 ) } v a r C i e = { a n i m a t i o n I t e r a t i o n C o u n t : 1 , b o r d e r I m a g e O u t s e t : 1 , b o r d e r I m a g e S l i c e : 1 , b o r d e r I m a g e W i d t h : 1 , b o x F l e x : 1 , b o x F l e x G r o u p : 1 , b o x O r d i n a l G r o u p : 1 , c o l u m n C o u n t : 1 , c o l u m n s : 1 , f l e x : 1 , f l e x G r o w : 1 , f l e x P o s i t i v e : 1 , f l e x S h r i n k : 1 , f l e x N e g a t i v e : 1 , f l e x O r d e r : 1 , g r i d R o w : 1 , g r i d R o w E n d : 1 , g r i d R o w S p a n : 1 , g r i d R o w S t a r t : 1 , g r i d C o l u m n : 1 , g r i d C o l u m n E n d : 1 , g r i d C o l u m n S p a n : 1 , g r i d C o l u m n S t a r t : 1 , m s G r i d R o w : 1 , m s G r i d R o w S p a n : 1 , m s G r i d C o l u m n : 1 , m s G r i d C o l u m n S p a n : 1 , f o n t W e i g h t : 1 , l i n e H e i g h t : 1 , o p a c i t y : 1 , o r d e r : 1 , o r p h a n s : 1 , t a b S i z e : 1 , w i d o w s : 1 , z I n d e x : 1 , z o o m : 1 , W e b k i t L i n e C l a m p : 1 , f i l l O p a c i t y : 1 , f l o o d O p a c i t y : 1 , s t o p O p a c i t y : 1 , s t r o k e D a s h a r r a y : 1 , s t r o k e D a s h o f f s e t : 1 , s t r o k e M i t e r l i m i t : 1 , s t r o k e
2022-12-26 15:23:16 +00:00
45 deg ,
$ { t } 25 % ,
transparent 25 % ,
transparent 50 % ,
$ { t } 50 % ,
$ { t } 75 % ,
transparent 75 % ,
transparent
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
) ` ,backgroundSize: ` $ { e } $ { e } ` }}function Yoe(e){const t=FF().toHexString();return!e||Woe(e)?t:e.string&&e.colors?Xoe(e.string,e.colors):e.string&&!e.colors?Koe(e.string):e.colors&&!e.string?Zoe(e.colors):t}function Koe(e){let t=0;if(e.length===0)return t.toString();for(let r=0;r<e.length;r+=1)t=e.charCodeAt(r)+((t<<5)-t),t=t&t;let n="#";for(let r=0;r<3;r+=1){const i=t>>r*8&255;n+= ` 00 $ { i . toString ( 16 ) } ` .substr(-2)}return n}function Xoe(e,t){let n=0;if(e.length===0)return t[0];for(let r=0;r<e.length;r+=1)n=e.charCodeAt(r)+((n<<5)-n),n=n&n;return n=(n%t.length+t.length)%t.length,t[n]}function Zoe(e){return e[Math.floor(Math.random()*e.length)]}function Tt(e,t){return n=>n.colorMode==="dark"?t:e}function sk(e){const{orientation:t,vertical:n,horizontal:r}=e;return t?t==="vertical"?n:r:{}}function Qoe(e){const t=typeof e;return e!=null&&(t==="object"||t==="function")&&!Array.isArray(e)}function VF(e){return Qoe(e)&&e.reference?e.reference:String(e)}var zS=(e,...t)=>t.map(VF).join( ` $ { e } ` ).replace(/calc/g,""),MA=(...e)=> ` calc ( $ { zS ( "+" , ... e ) } ) ` ,RA=(...e)=> ` calc ( $ { zS ( "-" , ... e ) } ) ` ,Z7=(...e)=> ` calc ( $ { zS ( "*" , ... e ) } ) ` ,IA=(...e)=> ` calc ( $ { zS ( "/" , ... e ) } ) ` ,DA=e=>{const t=VF(e);return t!=null&&!Number.isNaN(parseFloat(t))?String(t).startsWith("-")?String(t).slice(1): ` - $ { t } ` :Z7(t,-1)},qu=Object.assign(e=>({add:(...t)=>qu(MA(e,...t)),subtract:(...t)=>qu(RA(e,...t)),multiply:(...t)=>qu(Z7(e,...t)),divide:(...t)=>qu(IA(e,...t)),negate:()=>qu(DA(e)),toString:()=>e.toString()}),{add:MA,subtract:RA,multiply:Z7,divide:IA,negate:DA});function Joe(e){return!Number.isInteger(parseFloat(e.toString()))}function eae(e,t="-"){return e.replace(/ \s +/g,t)}function UF(e){const t=eae(e.toString());return t.includes(" \\ .")?e:Joe(e)?t.replace("."," \\ ."):e}function tae(e,t=""){return[t,UF(e)].filter(Boolean).join("-")}function nae(e,t){return ` var ( $ { UF ( e ) } $ { t ? ` , ${ t } ` : "" } ) ` }function rae(e,t=""){return ` -- $ { tae ( e , t ) } ` }function yi(e,t){const n=rae(e,t==null?void 0:t.prefix);return{variable:n,reference:nae(n,iae(t==null?void 0:t.fallback))}}function iae(e){return typeof e=="string"?e:e==null?void 0:e.reference}var{definePartsStyle:oae,defineMultiStyleConfig:aae}=lr(noe.keys),sae={borderTopWidth:"1px",borderColor:"inherit",_last:{borderBottomWidth:"1px"}},lae={transitionProperty:"common",transitionDuration:"normal",fontSize:"md",_focusVisible:{boxShadow:"outline"},_hover:{bg:"blackAlpha.50"},_disabled:{opacity:.4,cursor:"not-allowed"},px:"4",py:"2"},uae={pt:"2",px:"4",pb:"5"},cae={fontSize:"1.25em"},dae=oae({container:sae,button:lae,panel:uae,icon:cae}),fae=aae({baseStyle:dae}),{definePartsStyle:fy,defineMultiStyleConfig:hae}=lr(roe.keys),Oa=jn("alert-fg"),oc=jn("alert-bg"),pae=fy({container:{bg:oc.reference,px:"4",py:"3"},title:{fontWeight:"bold",lineHeight:"6",marginEnd:"2"},description:{lineHeight:"6"},icon:{color:Oa.reference,flexShrink:0,marginEnd:"3",w:"5",h:"6"},spinner:{color:Oa.reference,flexShrink:0,marginEnd:"3",w:"5",h:"5"}});function lk(e){const{theme:t,colorScheme:n}=e,r=Yg( ` $ { n } . 200 ` ,.16)(t);return{light: ` colors . $ { n } . 100 ` ,dark:r}}var gae=fy(e=>{const{colorScheme:t}=e,n=lk(e);return{container:{[Oa.variable]: ` colors . $ { t } . 500 ` ,[oc.variable]:n.light,_dark:{[Oa.variable]: ` colors . $ { t } . 200 ` ,[oc.variable]:n.dark}}}}),mae=fy(e=>{const{colorScheme:t}=e,n=lk(e);return{container:{[Oa.variable]: ` colors . $ { t } . 500 ` ,[oc.variable]:n.light,_dark:{[Oa.variable]: ` colors . $ { t } . 200 ` ,[oc.variable]:n.dark},paddingStart:"3",borderStartWidth:"4px",borderStartColor:Oa.reference}}}),vae=fy(e=>{const{colorScheme:t}=e,n=lk(e);return{container:{[Oa.variable]: ` colors . $ { t } . 500 ` ,[oc.variable]:n.light,_dark:{[Oa.variable]: ` colors . $ { t } . 200 ` ,[oc.variable]:n.dark},pt:"2",borderTopWidth:"4px",borderTopColor:Oa.reference}}}),yae=fy(e=>{const{colorScheme:t}=e;return{container:{[Oa.variable]:"colors.white",[oc.variable]: ` colors . $ { t } . 500 ` ,_dark:{[Oa.variable]:"colors.gray.900",[oc.variable]: ` colors . $ { t } . 200 ` },color:Oa.reference}}}),bae={subtle:gae,"left-accent":mae,"top-accent":vae,solid:yae},Sae=hae({baseStyle:pae,variants:bae,defaultProps:{variant:"subtle",colorScheme:"blue"}}),WF={px:"1px",.
2022-12-26 15:23:16 +00:00
to right ,
transparent 0 % ,
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
$ { Eo ( n , a ) } 50 % ,
2022-12-26 15:23:16 +00:00
transparent 100 %
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
) ` ;return{...!r&&i&&o,...r?{bgImage:s}:{bgColor:a}}},gue={lineHeight:"1",fontSize:"0.25em",fontWeight:"bold",color:"white"},mue=e=>({bg:Tt("gray.100","whiteAlpha.300")(e)}),vue=e=>({transitionProperty:"common",transitionDuration:"slow",...pue(e)}),yue=bv(e=>({label:gue,filledTrack:vue(e),track:mue(e)})),bue={xs:bv({track:{h:"1"}}),sm:bv({track:{h:"2"}}),md:bv({track:{h:"3"}}),lg:bv({track:{h:"4"}})},Sue=hue({sizes:bue,baseStyle:yue,defaultProps:{size:"md",colorScheme:"blue"}}),{defineMultiStyleConfig:xue,definePartsStyle:l4}=lr(yoe.keys),wue=e=>{var t;const n=(t=Lo(l5.baseStyle,e))==null?void 0:t.control;return{...n,borderRadius:"full",_checked:{...n==null?void 0:n._checked,_before:{content:'""',display:"inline-block",pos:"relative",w:"50%",h:"50%",borderRadius:"50%",bg:"currentColor"}}}},Cue=l4(e=>{var t,n,r,i;return{label:(n=(t=l5).baseStyle)==null?void 0:n.call(t,e).label,container:(i=(r=l5).baseStyle)==null?void 0:i.call(r,e).container,control:wue(e)}}),_ue={md:l4({control:{w:"4",h:"4"},label:{fontSize:"md"}}),lg:l4({control:{w:"5",h:"5"},label:{fontSize:"lg"}}),sm:l4({control:{width:"3",height:"3"},label:{fontSize:"sm"}})},kue=xue({baseStyle:Cue,sizes:_ue,defaultProps:{size:"md",colorScheme:"blue"}}),{defineMultiStyleConfig:Eue,definePartsStyle:Pue}=lr(boe.keys),J3=jn("select-bg"),zA,Tue={...(zA=Cn.baseStyle)==null?void 0:zA.field,appearance:"none",paddingBottom:"1px",lineHeight:"normal",bg:J3.reference,[J3.variable]:"colors.white",_dark:{[J3.variable]:"colors.gray.700"},"> option, > optgroup":{bg:J3.reference}},Lue={width:"6",height:"100%",insetEnd:"2",position:"relative",color:"currentColor",fontSize:"xl",_disabled:{opacity:.5}},Aue=Pue({field:Tue,icon:Lue}),eb={paddingInlineEnd:"8"},jA,HA,VA,UA,WA,GA,qA,YA,Oue={lg:{...(jA=Cn.sizes)==null?void 0:jA.lg,field:{...(HA=Cn.sizes)==null?void 0:HA.lg.field,...eb}},md:{...(VA=Cn.sizes)==null?void 0:VA.md,field:{...(UA=Cn.sizes)==null?void 0:UA.md.field,...eb}},sm:{...(WA=Cn.sizes)==null?void 0:WA.sm,field:{...(GA=Cn.sizes)==null?void 0:GA.sm.field,...eb}},xs:{...(qA=Cn.sizes)==null?void 0:qA.xs,field:{...(YA=Cn.sizes)==null?void 0:YA.xs.field,...eb},icon:{insetEnd:"1"}}},Mue=Eue({baseStyle:Aue,sizes:Oue,variants:Cn.variants,defaultProps:Cn.defaultProps}),C6=jn("skeleton-start-color"),_6=jn("skeleton-end-color"),Rue={[C6.variable]:"colors.gray.100",[_6.variable]:"colors.gray.400",_dark:{[C6.variable]:"colors.gray.800",[_6.variable]:"colors.gray.600"},background:C6.reference,borderColor:_6.reference,opacity:.7,borderRadius:"sm"},Iue={baseStyle:Rue},k6=jn("skip-link-bg"),Due={borderRadius:"md",fontWeight:"semibold",_focusVisible:{boxShadow:"outline",padding:"4",position:"fixed",top:"6",insetStart:"6",[k6.variable]:"colors.white",_dark:{[k6.variable]:"colors.gray.700"},bg:k6.reference}},Nue={baseStyle:Due},{defineMultiStyleConfig:Bue,definePartsStyle:jS}=lr(Soe.keys),R2=jn("slider-thumb-size"),I2=jn("slider-track-size"),Cd=jn("slider-bg"), $ ue=e=>{const{orientation:t}=e;return{display:"inline-block",position:"relative",cursor:"pointer",_disabled:{opacity:.6,cursor:"default",pointerEvents:"none"},...sk({orientation:t,vertical:{h:"100%"},horizontal:{w:"100%"}})}},Fue=e=>({...sk({orientation:e.orientation,horizontal:{h:I2.reference},vertical:{w:I2.reference}}),overflow:"hidden",borderRadius:"sm",[Cd.variable]:"colors.gray.200",_dark:{[Cd.variable]:"colors.whiteAlpha.200"},_disabled:{[Cd.variable]:"colors.gray.300",_dark:{[Cd.variable]:"colors.whiteAlpha.300"}},bg:Cd.reference}),zue=e=>{const{orientation:t}=e;return{...sk({orientation:t,vertical:{left:"50%",transform:"translateX(-50%)",_active:{transform:"translateX(-50%) scale(1.15)"}},horizontal:{top:"50%",transform:"translateY(-50%)",_active:{transform:"translateY(-50%) scale(1.15)"}}}),w:R2.reference,h:R2.reference,display:"flex",alignItems:"center",justifyContent:"center",position:"absolute",outline:0,zIndex:1,borderRadius:"full",bg:"white",boxShadow:"base",border:"1px solid",borderColor:"transparent",transitionProperty:"transform",transitionDuration:"normal",_focusVisible:{boxShadow:"outline"},_disabled:{bg:"gray.300"}}},
2022-12-26 15:23:16 +00:00
[ data - motion - pop - id = "${n}" ] {
position : absolute ! important ;
width : $ { o } px ! important ;
height : $ { a } px ! important ;
top : $ { s } px ! important ;
left : $ { l } px ! important ;
}
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` ),()=>{document.head.removeChild(u)}},[t]),y(Tge,{isPresent:t,childRef:r,sizeRef:i,children:C.exports.cloneElement(e,{ref:r})})}const F6=({children:e,initial:t,isPresent:n,onExitComplete:r,custom:i,presenceAffectsLayout:o,mode:a})=>{const s=GS(Age),l=C.exports.useId(),u=C.exports.useMemo(()=>({id:l,initial:t,isPresent:n,custom:i,onExitComplete:f=>{s.set(f,!0);for(const p of s.values())if(!p)return;r&&r()},register:f=>(s.set(f,!1),()=>s.delete(f))}),o?void 0:[n]);return C.exports.useMemo(()=>{s.forEach((f,p)=>s.set(p,!1))},[n]),C.exports.useEffect(()=>{!n&&!s.size&&r&&r()},[n]),a==="popLayout"&&(e=y(Lge,{isPresent:n,children:e})),y(p1.Provider,{value:u,children:e})};function Age(){return new Map}const $ 0=e=>e.key||"";function Oge(e,t){e.forEach(n=>{const r= $ 0(n);t.set(r,n)})}function Mge(e){const t=[];return C.exports.Children.forEach(e,n=>{C.exports.isValidElement(n)&&t.push(n)}),t}const cf=({children:e,custom:t,initial:n=!0,onExitComplete:r,exitBeforeEnter:i,presenceAffectsLayout:o=!0,mode:a="sync"})=>{i&&(a="wait",tj(!1,"Replace exitBeforeEnter with mode='wait'"));let[s]=Pge();const l=C.exports.useContext(fk).forceRender;l&&(s=l);const u=Lj(),f=Mge(e);let p=f;const m=new Set,v=C.exports.useRef(p),b=C.exports.useRef(new Map).current,x=C.exports.useRef(!0);if(d5(()=>{x.current=!1,Oge(f,b),v.current=p}),xk(()=>{x.current=!0,b.clear(),m.clear()}),x.current)return y(fn,{children:p.map(T=>y(F6,{isPresent:!0,initial:n?void 0:!1,presenceAffectsLayout:o,mode:a,children:T}, $ 0(T)))});p=[...p];const E=v.current.map( $ 0),k=f.map( $ 0),_=E.length;for(let T=0;T<_;T++){const A=E[T];k.indexOf(A)===-1&&m.add(A)}return a==="wait"&&m.size&&(p=[]),m.forEach(T=>{if(k.indexOf(T)!==-1)return;const A=b.get(T);if(!A)return;const R=E.indexOf(T),I=()=>{b.delete(T),m.delete(T);const D=v.current.findIndex(B=>B.key===T);if(v.current.splice(D,1),!m.size){if(v.current=f,u.current===!1)return;s(),r&&r()}};p.splice(R,0,y(F6,{isPresent:!1,onExitComplete:I,custom:t,presenceAffectsLayout:o,mode:a,children:A}, $ 0(A)))}),p=p.map(T=>{const A=T.key;return m.has(A)?T:y(F6,{isPresent:!0,presenceAffectsLayout:o,mode:a,children:T}, $ 0(T))}),ej!=="production"&&a==="wait"&&p.length>1&&console.warn( ` You ' re attempting to animate multiple children within AnimatePresence , but its mode is set to "wait" . This will lead to odd visual behaviour . ` ),y(fn,{children:m.size?p:p.map(T=>C.exports.cloneElement(T))})};var Ul=function(){return Ul=Object.assign||function(t){for(var n,r=1,i=arguments.length;r<i;r++){n=arguments[r];for(var o in n)Object.prototype.hasOwnProperty.call(n,o)&&(t[o]=n[o])}return t},Ul.apply(this,arguments)};function Aj(e,t){var n={};for(var r in e)Object.prototype.hasOwnProperty.call(e,r)&&t.indexOf(r)<0&&(n[r]=e[r]);if(e!=null&&typeof Object.getOwnPropertySymbols=="function")for(var i=0,r=Object.getOwnPropertySymbols(e);i<r.length;i++)t.indexOf(r[i])<0&&Object.prototype.propertyIsEnumerable.call(e,r[i])&&(n[r[i]]=e[r[i]]);return n}function m1(e,t,n,r){function i(o){return o instanceof n?o:new n(function(a){a(o)})}return new(n||(n=Promise))(function(o,a){function s(f){try{u(r.next(f))}catch(p){a(p)}}function l(f){try{u(r.throw(f))}catch(p){a(p)}}function u(f){f.done?o(f.value):i(f.value).then(s,l)}u((r=r.apply(e,t||[])).next())})}function v1(e,t){var n={label:0,sent:function(){if(o[0]&1)throw o[1];return o[1]},trys:[],ops:[]},r,i,o,a;return a={next:s(0),throw:s(1),return:s(2)},typeof Symbol=="function"&&(a[Symbol.iterator]=function(){return this}),a;function s(u){return function(f){return l([u,f])}}function l(u){if(r)throw new TypeError("Generator is already executing.");for(;a&&(a=0,u[0]&&(n=0)),n;)try{if(r=1,i&&(o=u[0]&2?i.return:u[0]?i.throw||((o=i.return)&&o.call(i),0):i.next)&&!(o=o.call(i,u[1])).done)return o;switch(i=0,o&&(u=[u[0]&2,o.value]),u[0]){case 0:case 1:o=u;break;case 4:return n.label++,{value:u[1],done:!1};case 5:n.label++,i=u[1],u=[0];continue;case 7:u=n.ops.pop(),n.trys.pop();continue;default:if(o=n.trys,!(o=o.length>0&&o[o.length-1])&&(u[0]===6||u[0]===2)){n=0;continue}if(u[0]===3&&(!o||u[1]>o[0]&&u[1]<o[3])){n.label=u[1];break}if(u[
` })}function n1e(e){JS({condition:e.isOpen&&!!e.isDisabled,message:"Cannot open a disabled accordion item"})}function ng(e){const{isOpen:t,isDisabled:n}=Bk(),{reduceMotion:r}= $ k(),i=yy("chakra-accordion__icon",e.className),o=ex(),a={opacity:n?.4:1,transform:t?"rotate(-180deg)":void 0,transition:r?void 0:"transform 0.2s",transformOrigin:"center",...o.icon};return y(za,{viewBox:"0 0 24 24","aria-hidden":!0,className:i,__css:a,...e,children:y("path",{fill:"currentColor",d:"M16.59 8.59L12 13.17 7.41 8.59 6 10l6 6 6-6z"})})}ng.displayName="AccordionIcon";var rg=Oe(function(t,n){const{children:r,className:i}=t,{htmlProps:o,...a}=Qge(t),l={...ex().container,overflowAnchor:"none"},u=C.exports.useMemo(()=>a,[a]);return oe.createElement(Gge,{value:u},oe.createElement(_e.div,{ref:n,...o,className:yy("chakra-accordion__item",i),__css:l},typeof r=="function"?r({isExpanded:!!a.isOpen,isDisabled:!!a.isDisabled}):r))});rg.displayName="AccordionItem";var ig=Oe(function(t,n){const{className:r,motionProps:i,...o}=t,{reduceMotion:a}= $ k(),{getPanelProps:s,isOpen:l}=Bk(),u=s(o,n),f=yy("chakra-accordion__panel",r),p=ex();a||delete u.hidden;const m=oe.createElement(_e.div,{...u,__css:p.panel,className:f});return a?m:y(Oj,{in:l,...i,children:m})});ig.displayName="AccordionPanel";var Fk=Oe(function({children:t,reduceMotion:n,...r},i){const o=Xi("Accordion",r),a=_n(r),{htmlProps:s,descendants:l,...u}=Xge(a),f=C.exports.useMemo(()=>({...u,reduceMotion:!!n}),[u,n]);return oe.createElement(qge,{value:l},oe.createElement(Zge,{value:f},oe.createElement(Wge,{value:o},oe.createElement(_e.div,{ref:i,...s,className:yy("chakra-accordion",r.className),__css:o.root},t))))});Fk.displayName="Accordion";var r1e=(...e)=>e.filter(Boolean).join(" "),i1e=uf({"0%":{transform:"rotate(0deg)"},"100%":{transform:"rotate(360deg)"}}),by=Oe((e,t)=>{const n=Ro("Spinner",e),{label:r="Loading...",thickness:i="2px",speed:o="0.45s",emptyColor:a="transparent",className:s,...l}=_n(e),u=r1e("chakra-spinner",s),f={display:"inline-block",borderColor:"currentColor",borderStyle:"solid",borderRadius:"99999px",borderWidth:i,borderBottomColor:a,borderLeftColor:a,animation: ` $ { i1e } $ { o } linear infinite ` ,...n};return oe.createElement(_e.div,{ref:t,__css:f,className:u,...l},r&&oe.createElement(_e.span,{srOnly:!0},r))});by.displayName="Spinner";var tx=(...e)=>e.filter(Boolean).join(" ");function o1e(e){return y(za,{viewBox:"0 0 24 24",...e,children:y("path",{fill:"currentColor",d:"M12,0A12,12,0,1,0,24,12,12.014,12.014,0,0,0,12,0Zm6.927,8.2-6.845,9.289a1.011,1.011,0,0,1-1.43.188L5.764,13.769a1,1,0,1,1,1.25-1.562l4.076,3.261,6.227-8.451A1,1,0,1,1,18.927,8.2Z"})})}function a1e(e){return y(za,{viewBox:"0 0 24 24",...e,children:y("path",{fill:"currentColor",d:"M12,0A12,12,0,1,0,24,12,12.013,12.013,0,0,0,12,0Zm.25,5a1.5,1.5,0,1,1-1.5,1.5A1.5,1.5,0,0,1,12.25,5ZM14.5,18.5h-4a1,1,0,0,1,0-2h.75a.25.25,0,0,0,.25-.25v-4.5a.25.25,0,0,0-.25-.25H10.5a1,1,0,0,1,0-2h1a2,2,0,0,1,2,2v4.75a.25.25,0,0,0,.25.25h.75a1,1,0,1,1,0,2Z"})})}function pM(e){return y(za,{viewBox:"0 0 24 24",...e,children:y("path",{fill:"currentColor",d:"M11.983,0a12.206,12.206,0,0,0-8.51,3.653A11.8,11.8,0,0,0,0,12.207,11.779,11.779,0,0,0,11.8,24h.214A12.111,12.111,0,0,0,24,11.791h0A11.766,11.766,0,0,0,11.983,0ZM10.5,16.542a1.476,1.476,0,0,1,1.449-1.53h.027a1.527,1.527,0,0,1,1.523,1.47,1.475,1.475,0,0,1-1.449,1.53h-.027A1.529,1.529,0,0,1,10.5,16.542ZM11,12.5v-6a1,1,0,0,1,2,0v6a1,1,0,1,1-2,0Z"})})}var[s1e,l1e]=An({name:"AlertContext",hookName:"useAlertContext",providerName:"<Alert />"}),[u1e,zk]=An({name:"AlertStylesContext",hookName:"useAlertStyles",providerName:"<Alert />"}),Dj={info:{icon:a1e,colorScheme:"blue"},warning:{icon:pM,colorScheme:"orange"},success:{icon:o1e,colorScheme:"green"},error:{icon:pM,colorScheme:"red"},loading:{icon:by,colorScheme:"blue"}};function c1e(e){return Dj[e].colorScheme}function d1e(e){return Dj[e].icon}var Nj=Oe(function(t,n){var u;const{status:r="info",addRole:i=!0,...o}=_n(t),a=(u=t.colorScheme)!=null?u:c1e(r),s=Xi("Alert",{...t,colorScheme:a}),l={width:"100%",display:"flex",alignItems:"center",positio
2022-12-26 15:23:16 +00:00
: root {
-- chakra - vh : 100 vh ;
}
@ supports ( height : - webkit - fill - available ) {
: root {
-- chakra - vh : - webkit - fill - available ;
}
}
@ supports ( height : - moz - fill - available ) {
: root {
-- chakra - vh : - moz - fill - available ;
}
}
@ supports ( height : 100 dvh ) {
: root {
-- chakra - vh : 100 dvh ;
}
}
2022-12-28 18:25:07 +00:00
` ,q1e=()=>y( $ S,{styles:Gj}),Y1e=()=>y( $ S,{styles: `
2022-12-26 15:23:16 +00:00
html {
line - height : 1.5 ;
- webkit - text - size - adjust : 100 % ;
font - family : system - ui , sans - serif ;
- webkit - font - smoothing : antialiased ;
text - rendering : optimizeLegibility ;
- moz - osx - font - smoothing : grayscale ;
touch - action : manipulation ;
}
body {
position : relative ;
min - height : 100 % ;
font - feature - settings : 'kern' ;
}
* ,
* : : before ,
* : : after {
border - width : 0 ;
border - style : solid ;
box - sizing : border - box ;
}
main {
display : block ;
}
hr {
border - top - width : 1 px ;
box - sizing : content - box ;
height : 0 ;
overflow : visible ;
}
pre ,
code ,
kbd ,
samp {
font - family : SFMono - Regular , Menlo , Monaco , Consolas , monospace ;
font - size : 1 em ;
}
a {
background - color : transparent ;
color : inherit ;
text - decoration : inherit ;
}
abbr [ title ] {
border - bottom : none ;
text - decoration : underline ;
- webkit - text - decoration : underline dotted ;
text - decoration : underline dotted ;
}
b ,
strong {
font - weight : bold ;
}
small {
font - size : 80 % ;
}
sub ,
sup {
font - size : 75 % ;
line - height : 0 ;
position : relative ;
vertical - align : baseline ;
}
sub {
bottom : - 0.25 em ;
}
sup {
top : - 0.5 em ;
}
img {
border - style : none ;
}
button ,
input ,
optgroup ,
select ,
textarea {
font - family : inherit ;
font - size : 100 % ;
line - height : 1.15 ;
margin : 0 ;
}
button ,
input {
overflow : visible ;
}
button ,
select {
text - transform : none ;
}
button : : - moz - focus - inner ,
[ type = "button" ] : : - moz - focus - inner ,
[ type = "reset" ] : : - moz - focus - inner ,
[ type = "submit" ] : : - moz - focus - inner {
border - style : none ;
padding : 0 ;
}
fieldset {
padding : 0.35 em 0.75 em 0.625 em ;
}
legend {
box - sizing : border - box ;
color : inherit ;
display : table ;
max - width : 100 % ;
padding : 0 ;
white - space : normal ;
}
progress {
vertical - align : baseline ;
}
textarea {
overflow : auto ;
}
[ type = "checkbox" ] ,
[ type = "radio" ] {
box - sizing : border - box ;
padding : 0 ;
}
[ type = "number" ] : : - webkit - inner - spin - button ,
[ type = "number" ] : : - webkit - outer - spin - button {
- webkit - appearance : none ! important ;
}
input [ type = "number" ] {
- moz - appearance : textfield ;
}
[ type = "search" ] {
- webkit - appearance : textfield ;
outline - offset : - 2 px ;
}
[ type = "search" ] : : - webkit - search - decoration {
- webkit - appearance : none ! important ;
}
: : - webkit - file - upload - button {
- webkit - appearance : button ;
font : inherit ;
}
details {
display : block ;
}
summary {
display : list - item ;
}
template {
display : none ;
}
[ hidden ] {
display : none ! important ;
}
body ,
blockquote ,
dl ,
dd ,
h1 ,
h2 ,
h3 ,
h4 ,
h5 ,
h6 ,
hr ,
figure ,
p ,
pre {
margin : 0 ;
}
button {
background : transparent ;
padding : 0 ;
}
fieldset {
margin : 0 ;
padding : 0 ;
}
ol ,
ul {
margin : 0 ;
padding : 0 ;
}
textarea {
resize : vertical ;
}
button ,
[ role = "button" ] {
cursor : pointer ;
}
button : : - moz - focus - inner {
border : 0 ! important ;
}
table {
border - collapse : collapse ;
}
h1 ,
h2 ,
h3 ,
h4 ,
h5 ,
h6 {
font - size : inherit ;
font - weight : inherit ;
}
button ,
input ,
optgroup ,
select ,
textarea {
padding : 0 ;
line - height : inherit ;
color : inherit ;
}
img ,
svg ,
video ,
canvas ,
audio ,
iframe ,
embed ,
object {
display : block ;
}
img ,
video {
max - width : 100 % ;
height : auto ;
}
[ data - js - focus - visible ] : focus : not ( [ data - focus - visible - added ] ) : not ( [ data - focus - visible - disabled ] ) {
outline : none ;
box - shadow : none ;
}
select : : - ms - expand {
display : none ;
}
2022-12-28 18:25:07 +00:00
$ { Gj }
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` });function jh(e,t,n,r){const i=wr(n);return C.exports.useEffect(()=>{const o=typeof e=="function"?e():e!=null?e:document;if(!(!n||!o))return o.addEventListener(t,i,r),()=>{o.removeEventListener(t,i,r)}},[t,e,r,i,n]),()=>{const o=typeof e=="function"?e():e!=null?e:document;o==null||o.removeEventListener(t,i,r)}}function K1e(e){return"current"in e}var qj=()=>typeof window<"u";function X1e(){var t;const e=navigator.userAgentData;return(t=e==null?void 0:e.platform)!=null?t:navigator.platform}var Z1e=e=>qj()&&e.test(navigator.vendor),Q1e=e=>qj()&&e.test(X1e()),J1e=()=>Q1e(/mac|iphone|ipad|ipod/i),eme=()=>J1e()&&Z1e(/apple/i);function tme(e){const{ref:t,elements:n,enabled:r}=e,i=()=>{var a;var o;return(a=(o=t.current)==null?void 0:o.ownerDocument)!=null?a:document};jh(i,"pointerdown",o=>{if(!eme()||!r)return;const a=o.target,l=(n!=null?n:[t]).some(u=>{const f=K1e(u)?u.current:u;return(f==null?void 0:f.contains(a))||f===a});i().activeElement!==a&&l&&(o.preventDefault(),a.focus())})}var nme=Gie?C.exports.useLayoutEffect:C.exports.useEffect;function wM(e,t=[]){const n=C.exports.useRef(e);return nme(()=>{n.current=e}),C.exports.useCallback((...r)=>{var i;return(i=n.current)==null?void 0:i.call(n,...r)},t)}function rme(e,t){const n=e!==void 0;return[n,n&&typeof e<"u"?e:t]}function ime(e,t){const n=C.exports.useId();return C.exports.useMemo(()=>e||[t,n].filter(Boolean).join("-"),[e,t,n])}function Qh(e={}){const{onClose:t,onOpen:n,isOpen:r,id:i}=e,o=wM(n),a=wM(t),[s,l]=C.exports.useState(e.defaultIsOpen||!1),[u,f]=rme(r,s),p=ime(i,"disclosure"),m=C.exports.useCallback(()=>{u||l(!1),a==null||a()},[u,a]),v=C.exports.useCallback(()=>{u||l(!0),o==null||o()},[u,o]),b=C.exports.useCallback(()=>{(f?m:v)()},[f,v,m]);return{isOpen:!!f,onOpen:v,onClose:m,onToggle:b,isControlled:u,getButtonProps:(x={})=>({...x,"aria-expanded":f,"aria-controls":p,onClick:qie(x.onClick,b)}),getDisclosureProps:(x={})=>({...x,hidden:!f,id:p})}}function Gk(e){const t=Object.assign({},e);for(let n in t)t[n]===void 0&&delete t[n];return t}var qk=Oe(function(t,n){const{htmlSize:r,...i}=t,o=Xi("Input",i),a=_n(i),s=jk(a),l=Xr("chakra-input",t.className);return oe.createElement(_e.input,{size:r,...s,__css:o.field,ref:n,className:l})});qk.displayName="Input";qk.id="Input";var[ome,Yj]=An({name:"InputGroupStylesContext",errorMessage: ` useInputGroupStyles returned is 'undefined' . Seems you forgot to wrap the components in "<InputGroup />" ` }),ame=Oe(function(t,n){const r=Xi("Input",t),{children:i,className:o,...a}=_n(t),s=Xr("chakra-input__group",o),l={},u=rx(i),f=r.field;u.forEach(m=>{var v,b;!r||(f&&m.type.id==="InputLeftElement"&&(l.paddingStart=(v=f.height)!=null?v:f.h),f&&m.type.id==="InputRightElement"&&(l.paddingEnd=(b=f.height)!=null?b:f.h),m.type.id==="InputRightAddon"&&(l.borderEndRadius=0),m.type.id==="InputLeftAddon"&&(l.borderStartRadius=0))});const p=u.map(m=>{var v,b;const x=Gk({size:((v=m.props)==null?void 0:v.size)||t.size,variant:((b=m.props)==null?void 0:b.variant)||t.variant});return m.type.id!=="Input"?C.exports.cloneElement(m,x):C.exports.cloneElement(m,Object.assign(x,l,m.props))});return oe.createElement(_e.div,{className:s,ref:n,__css:{width:"100%",display:"flex",position:"relative"},...a},y(ome,{value:r,children:p}))});ame.displayName="InputGroup";var sme={left:{marginEnd:"-1px",borderEndRadius:0,borderEndColor:"transparent"},right:{marginStart:"-1px",borderStartRadius:0,borderStartColor:"transparent"}},lme=_e("div",{baseStyle:{flex:"0 0 auto",width:"auto",display:"flex",alignItems:"center",whiteSpace:"nowrap"}}),Yk=Oe(function(t,n){var s;const{placement:r="left",...i}=t,o=(s=sme[r])!=null?s:{},a=Yj();return y(lme,{ref:n,...i,__css:{...a.addon,...o}})});Yk.displayName="InputAddon";var Kj=Oe(function(t,n){return y(Yk,{ref:n,placement:"left",...t,className:Xr("chakra-input__left-addon",t.className)})});Kj.displayName="InputLeftAddon";Kj.id="InputLeftAddon";var Xj=Oe(function(t,n){return y(Yk,{ref:n,placement:"right",...t,className:Xr("chakra-input__right-addon",t.className)})});Xj.displayName="InputRightAddon";Xj.id="InputRightAddon";var ume=_e("
2022-12-28 18:25:07 +00:00
. ` .concat(Dye, ` {
2022-12-26 15:23:16 +00:00
overflow : hidden ` ).concat(r, ` ;
padding - right : ` ).concat(s,"px ").concat(r, ` ;
}
body {
overflow : hidden ` ).concat(r, ` ;
overscroll - behavior : contain ;
` ).concat([t&&"position: relative ".concat(r,";"),n==="margin"&& `
padding - left : ` .concat(i, ` px ;
padding - top : ` ).concat(o, ` px ;
padding - right : ` ).concat(a, ` px ;
margin - left : 0 ;
margin - top : 0 ;
margin - right : ` ).concat(s,"px ").concat(r, ` ;
` ),n==="padding"&&"padding-right: ".concat(s,"px ").concat(r,";")].filter(Boolean).join(""), `
}
2022-12-27 17:21:02 +00:00
. ` ).concat(g4, ` {
2022-12-26 15:23:16 +00:00
right : ` ).concat(s,"px ").concat(r, ` ;
}
2022-12-27 17:21:02 +00:00
. ` ).concat(m4, ` {
2022-12-26 15:23:16 +00:00
margin - right : ` ).concat(s,"px ").concat(r, ` ;
}
2022-12-27 17:21:02 +00:00
. ` ).concat(g4," .").concat(g4, ` {
2022-12-26 15:23:16 +00:00
right : 0 ` ).concat(r, ` ;
}
2022-12-27 17:21:02 +00:00
. ` ).concat(m4," .").concat(m4, ` {
2022-12-26 15:23:16 +00:00
margin - right : 0 ` ).concat(r, ` ;
}
body {
2022-12-28 18:25:07 +00:00
` ).concat(Nye,": ").concat(s, ` px ;
2022-12-26 15:23:16 +00:00
}
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` )},Yye=function(e){var t=e.noRelative,n=e.noImportant,r=e.gapMode,i=r===void 0?"margin":r,o=C.exports.useMemo(function(){return Wye(i)},[i]);return y(Gye,{styles:qye(o,!t,i,n?"":"!important")})}, $ 9=!1;if(typeof window<"u")try{var fb=Object.defineProperty({},"passive",{get:function(){return $ 9=!0,!0}});window.addEventListener("test",fb,fb),window.removeEventListener("test",fb,fb)}catch(e){ $ 9=!1}var P0= $ 9?{passive:!1}:!1,Kye=function(e){return e.tagName==="TEXTAREA"},pV=function(e,t){var n=window.getComputedStyle(e);return n[t]!=="hidden"&&!(n.overflowY===n.overflowX&&!Kye(e)&&n[t]==="visible")},Xye=function(e){return pV(e,"overflowY")},Zye=function(e){return pV(e,"overflowX")},HM=function(e,t){var n=t;do{typeof ShadowRoot<"u"&&n instanceof ShadowRoot&&(n=n.host);var r=gV(e,n);if(r){var i=mV(e,n),o=i[1],a=i[2];if(o>a)return!0}n=n.parentNode}while(n&&n!==document.body);return!1},Qye=function(e){var t=e.scrollTop,n=e.scrollHeight,r=e.clientHeight;return[t,n,r]},Jye=function(e){var t=e.scrollLeft,n=e.scrollWidth,r=e.clientWidth;return[t,n,r]},gV=function(e,t){return e==="v"?Xye(t):Zye(t)},mV=function(e,t){return e==="v"?Qye(t):Jye(t)},e3e=function(e,t){return e==="h"&&t==="rtl"?-1:1},t3e=function(e,t,n,r,i){var o=e3e(e,window.getComputedStyle(t).direction),a=o*r,s=n.target,l=t.contains(s),u=!1,f=a>0,p=0,m=0;do{var v=mV(e,s),b=v[0],x=v[1],E=v[2],k=x-E-o*b;(b||k)&&gV(e,s)&&(p+=k,m+=b),s=s.parentNode}while(!l&&s!==document.body||l&&(t.contains(s)||t===s));return(f&&(i&&p===0||!i&&a>p)||!f&&(i&&m===0||!i&&-a>m))&&(u=!0),u},hb=function(e){return"changedTouches"in e?[e.changedTouches[0].clientX,e.changedTouches[0].clientY]:[0,0]},VM=function(e){return[e.deltaX,e.deltaY]},UM=function(e){return e&&"current"in e?e.current:e},n3e=function(e,t){return e[0]===t[0]&&e[1]===t[1]},r3e=function(e){return `
2022-12-26 15:23:16 +00:00
. block - interactivity - ` .concat(e, ` { pointer - events : none ; }
. allow - interactivity - ` ).concat(e, ` { pointer - events : all ; }
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` )},i3e=0,T0=[];function o3e(e){var t=C.exports.useRef([]),n=C.exports.useRef([0,0]),r=C.exports.useRef(),i=C.exports.useState(i3e++)[0],o=C.exports.useState(function(){return hV()})[0],a=C.exports.useRef(e);C.exports.useEffect(function(){a.current=e},[e]),C.exports.useEffect(function(){if(e.inert){document.body.classList.add("block-interactivity-".concat(i));var x=w9([e.lockRef.current],(e.shards||[]).map(UM),!0).filter(Boolean);return x.forEach(function(E){return E.classList.add("allow-interactivity-".concat(i))}),function(){document.body.classList.remove("block-interactivity-".concat(i)),x.forEach(function(E){return E.classList.remove("allow-interactivity-".concat(i))})}}},[e.inert,e.lockRef.current,e.shards]);var s=C.exports.useCallback(function(x,E){if("touches"in x&&x.touches.length===2)return!a.current.allowPinchZoom;var k=hb(x),_=n.current,T="deltaX"in x?x.deltaX:_[0]-k[0],A="deltaY"in x?x.deltaY:_[1]-k[1],R,I=x.target,D=Math.abs(T)>Math.abs(A)?"h":"v";if("touches"in x&&D==="h"&&I.type==="range")return!1;var B=HM(D,I);if(!B)return!0;if(B?R=D:(R=D==="v"?"h":"v",B=HM(D,I)),!B)return!1;if(!r.current&&"changedTouches"in x&&(T||A)&&(r.current=R),!R)return!0;var z=r.current||R;return t3e(z,E,x,z==="h"?T:A,!0)},[]),l=C.exports.useCallback(function(x){var E=x;if(!(!T0.length||T0[T0.length-1]!==o)){var k="deltaY"in E?VM(E):hb(E),_=t.current.filter(function(R){return R.name===E.type&&R.target===E.target&&n3e(R.delta,k)})[0];if(_&&_.should){E.cancelable&&E.preventDefault();return}if(!_){var T=(a.current.shards||[]).map(UM).filter(Boolean).filter(function(R){return R.contains(E.target)}),A=T.length>0?s(E,T[0]):!a.current.noIsolation;A&&E.cancelable&&E.preventDefault()}}},[]),u=C.exports.useCallback(function(x,E,k,_){var T={name:x,delta:E,target:k,should:_};t.current.push(T),setTimeout(function(){t.current=t.current.filter(function(A){return A!==T})},1)},[]),f=C.exports.useCallback(function(x){n.current=hb(x),r.current=void 0},[]),p=C.exports.useCallback(function(x){u(x.type,VM(x),x.target,s(x,e.lockRef.current))},[]),m=C.exports.useCallback(function(x){u(x.type,hb(x),x.target,s(x,e.lockRef.current))},[]);C.exports.useEffect(function(){return T0.push(o),e.setCallbacks({onScrollCapture:p,onWheelCapture:p,onTouchMoveCapture:m}),document.addEventListener("wheel",l,P0),document.addEventListener("touchmove",l,P0),document.addEventListener("touchstart",f,P0),function(){T0=T0.filter(function(x){return x!==o}),document.removeEventListener("wheel",l,P0),document.removeEventListener("touchmove",l,P0),document.removeEventListener("touchstart",f,P0)}},[]);var v=e.removeScrollBar,b=e.inert;return q(fn,{children:[b?y(o,{styles:r3e(i)}):null,v?y(Yye,{gapMode:"margin"}):null]})}const a3e=F2e(fV,o3e);var vV=C.exports.forwardRef(function(e,t){return y(ux,{...Ul({},e,{ref:t,sideCar:a3e})})});vV.classNames=ux.classNames;const yV=vV;var cp=(...e)=>e.filter(Boolean).join(" ");function Sv(...e){return function(n){e.some(r=>(r==null||r(n),n==null?void 0:n.defaultPrevented))}}var s3e=class{constructor(){sn(this,"modals");this.modals=[]}add(e){this.modals.push(e)}remove(e){this.modals=this.modals.filter(t=>t!==e)}isTopModal(e){return this.modals[this.modals.length-1]===e}},F9=new s3e;function l3e(e,t){C.exports.useEffect(()=>(t&&F9.add(e),()=>{F9.remove(e)}),[t,e])}function u3e(e){const{isOpen:t,onClose:n,id:r,closeOnOverlayClick:i=!0,closeOnEsc:o=!0,useInert:a=!0,onOverlayClick:s,onEsc:l}=e,u=C.exports.useRef(null),f=C.exports.useRef(null),[p,m,v]=d3e(r,"chakra-modal","chakra-modal--header","chakra-modal--body");c3e(u,t&&a),l3e(u,t);const b=C.exports.useRef(null),x=C.exports.useCallback(B=>{b.current=B.target},[]),E=C.exports.useCallback(B=>{B.key==="Escape"&&(B.stopPropagation(),o&&(n==null||n()),l==null||l())},[o,n,l]),[k,_]=C.exports.useState(!1),[T,A]=C.exports.useState(!1),R=C.exports.useCallback((B={},z=null)=>({role:"dialog",...B,ref:qn(z,u),id:p,tabIndex:-1,"aria-modal":!0,"aria-labelledby":k?m:void 0,"aria-describedby":T?v:void 0,onClick:Sv(B.onClick,j=>j.stopPropagation())}),[v,T,p,m,k]),I=C.exports.useCallback(B=>{B.stopPropagation(),b.cu
2022-12-26 15:23:16 +00:00
` ;if(!(0,e.isString)( $ )|| $ .length<1)throw new Error( ` $ { Z } Name ( key ) of reducer is required . $ { U } ` );if(!G||!G.length)return;const{duplicates:X,subsets:le}=(0,e.findDuplicatesAndSubsets)(G);if(X.length>1)throw new Error( ` $ { Z } Duplicated paths .
$ { JSON . stringify ( X ) }
$ { U } ` );if(le.length>1)throw new Error( ` $ { Z } You are trying to persist an entire property and also some of its subset .
$ { JSON . stringify ( le ) }
$ { U } ` )};e.singleTransformValidator=B;const z=function(G){if(!(0,e.isArray)(G))return;const $ =(G==null?void 0:G.map(V=>V.deepPersistKey).filter(V=>V))||[];if( $ .length){const V= $ .filter((J,Z)=> $ .indexOf(J)!==Z);if(V.length)throw new Error( ` $ { t . PACKAGE _NAME } : found duplicated keys in transforms creators . You can createWhitelist or createBlacklist for a specific root reducer key only once . Duplicated keys among createWhitelist and createBlacklist transforms are not allowed .
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
Duplicates : $ { JSON . stringify ( V ) } ` )}};e.transformsValidator=z;const j=function({whitelist:G,blacklist: $ }){if(G&&G.length&& $ && $ .length)throw new Error( ` $ { t . PACKAGE _NAME } : you should not define a whitelist and blacklist in parallel . It is allowed to use only one of these lists per config . ` );if(G){const{duplicates:V,subsets:J}=(0,e.findDuplicatesAndSubsets)(G);(0,e.throwError)({duplicates:V,subsets:J},"whitelist")}if( $ ){const{duplicates:V,subsets:J}=(0,e.findDuplicatesAndSubsets)( $ );(0,e.throwError)({duplicates:V,subsets:J},"blacklist")}};e.configValidator=j;const K=function({duplicates:G,subsets: $ },V){if(G.length)throw new Error( ` $ { t . PACKAGE _NAME } : duplicates of paths found in your $ { V } .
2022-12-26 15:23:16 +00:00
$ { JSON . stringify ( G ) } ` );if( $ .length)throw new Error( ` $ { t . PACKAGE _NAME } : subsets of some parent keys found in your $ { V } . You must decide if you want to persist an entire path or its specific subset .
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
$ { JSON . stringify ( $ ) } ` )};e.throwError=K;const te=function(G){return(0,e.isArray)(G)?G.filter(e.unique).reduce(( $ ,V)=>{const J=V.split("."),Z=J[0],U=J.slice(1).join(".")||void 0,X= $ .filter(fe=>Object.keys(fe)[0]===Z)[0],le=X?Object.values(X)[0]:void 0;return X|| $ .push({[Z]:U?[U]:void 0}),X&&!le&&U&&(X[Z]=[U]),X&&le&&U&&le.push(U), $ },[]):[]};e.getRootKeysGroup=te})(yU);(function(e){var t=_o&&_o.__rest||function(p,m){var v={};for(var b in p)Object.prototype.hasOwnProperty.call(p,b)&&m.indexOf(b)<0&&(v[b]=p[b]);if(p!=null&&typeof Object.getOwnPropertySymbols=="function")for(var x=0,b=Object.getOwnPropertySymbols(p);x<b.length;x++)m.indexOf(b[x])<0&&Object.prototype.propertyIsEnumerable.call(p,b[x])&&(v[b[x]]=p[b[x]]);return v};Object.defineProperty(e,"__esModule",{value:!0}),e.getPersistConfig=e.getTransforms=e.createBlacklist=e.createWhitelist=e.autoMergeDeep=void 0;const n=yU,r=np,i=jE,o=function(p,m,v={}){const b=v.whitelist||null,x=v.blacklist||null;function E(k){return!!(b&&b.indexOf(k)===-1||x&&x.indexOf(k)!==-1)}return{in:(k,_,T)=>!E(_)&&p?p(k,_,T):k,out:(k,_,T)=>!E(_)&&m?m(k,_,T):k,deepPersistKey:b&&b[0]}},a=(p,m,v,{debug:b,whitelist:x,blacklist:E,transforms:k})=>{if(x||E)throw new Error("State reconciler autoMergeDeep uses custom transforms instead of old whitelist or blacklist config properties. Please use createWhitelist or createBlacklist transforms.");(0,n.transformsValidator)(k);const _=(0,n.cloneDeep)(v);let T=p;if(T&&(0,n.isObjectLike)(T)){const A=(0,n.difference)(m,v);(0,n.isEmpty)(A)||(T=(0,n.mergeDeep)(p,A,{preserveUndefined:!0}),b&&console.log( ` $ { r . PACKAGE _NAME } / autoMergeDeep : sub state of your state was modified by reducer during rehydration . Values from reducer will be kept : $ { JSON . stringify ( A ) } ` )),Object.keys(T).forEach(R=>{if(R!=="_persist"){if((0,n.isObjectLike)(_[R])){_[R]=(0,n.mergeDeep)(_[R],T[R]);return}_[R]=T[R]}})}return b&&T&&(0,n.isObjectLike)(T)&&console.log( ` $ { r . PACKAGE _NAME } / autoMergeDeep : rehydrated keys $ { JSON . stringify ( T ) } ` ),_};e.autoMergeDeep=a;const s=(p,m)=>((0,n.singleTransformValidator)(m,p,i.ConfigType.WHITELIST),o(v=>{if(!m||!m.length)return v;let b=null,x;return m.forEach(E=>{const k=E.split(".");x=(0,n.path)(v,k),typeof x>"u"&&(0,n.isIntegerString)(k[k.length-1])&&(x=r.PLACEHOLDER_UNDEFINED);const _=(0,n.assocPath)(k,x),T=(0,n.isArray)(_)?[]:{};b=(0,n.mergeDeep)(b||T,_,{preservePlaceholder:!0})}),b||v},v=>(0,n.preserveUndefined)(v,m,i.ConfigType.WHITELIST),{whitelist:[p]}));e.createWhitelist=s;const l=(p,m)=>((0,n.singleTransformValidator)(m,p,i.ConfigType.BLACKLIST),o(v=>{if(!m||!m.length)return;const b=(0,n.preserveUndefined)(v,m,i.ConfigType.BLACKLIST,!0);return m.map(E=>E.split(".")).reduce((E,k)=>(0,n.dissocPath)(E,k),b)},v=>(0,n.preserveUndefined)(v,m,i.ConfigType.BLACKLIST),{whitelist:[p]}));e.createBlacklist=l;const u=function(p,m){return m.map(v=>{const b=Object.keys(v)[0],x=v[b];return p===i.ConfigType.WHITELIST?(0,e.createWhitelist)(b,x):(0,e.createBlacklist)(b,x)})};e.getTransforms=u;const f=p=>{var{key:m,whitelist:v,blacklist:b,storage:x,transforms:E,rootReducer:k}=p,_=t(p,["key","whitelist","blacklist","storage","transforms","rootReducer"]);(0,n.configValidator)({whitelist:v,blacklist:b});const T=(0,n.getRootKeysGroup)(v),A=(0,n.getRootKeysGroup)(b),R=Object.keys(k(void 0,{type:""})),I=T.map(te=>Object.keys(te)[0]),D=A.map(te=>Object.keys(te)[0]),B=R.filter(te=>I.indexOf(te)===-1&&D.indexOf(te)===-1),z=(0,e.getTransforms)(i.ConfigType.WHITELIST,T),j=(0,e.getTransforms)(i.ConfigType.BLACKLIST,A),K=(0,n.isArray)(v)?B.map(te=>(0,e.createBlacklist)(te)):[];return Object.assign(Object.assign({},_),{key:m,storage:x,transforms:[...z,...j,...K,...E||[]],stateReconciler:e.autoMergeDeep})};e.getPersistConfig=f})(vU);const x4=e=>e.length===1?e[0].prompt:e.map(t=> ` $ { t . prompt } : $ { t . weight } ` ).join(" "),G5e=e=>{const r=e.split(",").map(i=>i.split(":")).map(i=>({seed:Number(i[0]),weight:Number(i[1])}));return HE(r)?r:!1},HE=e=>Boolean(typeof e=="string"?G5e(e):e.length&&!e.some(t=>{const{seed:n,weight:r}=t,i=!isNaN(parseInt(n.toString(),10)),o=!isNaN(parseInt(r.toString(),10))&&r
2022-12-26 15:23:16 +00:00
* @ license
* Lodash < https : //lodash.com/>
* Copyright OpenJS Foundation and other contributors < https : //openjsf.org/>
* Released under MIT license < https : //lodash.com/license>
* Based on Underscore . js 1.8 . 3 < http : //underscorejs.org/LICENSE>
* Copyright Jeremy Ashkenas , DocumentCloud and Investigative Reporters & Editors
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
* /(function(e,t){(function(){var n,r="4.17.21",i=200,o="Unsupported core-js use. Try https:/ / npms . io / search ? q = ponyfill . ",a=" Expected a function ",s=" Invalid ` variable ` option passed into ` _.template ` ",l=" _ _lodash _hash _undefined _ _ ",u=500,f=" _ _lodash _placeholder _ _ ",p=1,m=2,v=4,b=1,x=2,E=1,k=2,_=4,T=8,A=16,R=32,I=64,D=128,B=256,z=512,j=30,K=" ... ",te=800,G=16,$=1,V=2,J=3,Z=1/0,U=9007199254740991,X=17976931348623157e292,le=0/0,fe=4294967295,Se=fe-1,Ae=fe>>>1,be=[[" ary ",D],[" bind ",E],[" bindKey ",k],[" curry ",T],[" curryRight ",A],[" flip ",z],[" partial ",R],[" partialRight ",I],[" rearg ",B]],ke=" [ object Arguments ] ",De=" [ object Array ] ",Je=" [ object AsyncFunction ] ",Ke=" [ object Boolean ] ",mt=" [ object Date ] ",_t=" [ object DOMException ] ",dt=" [ object Error ] ",vt=" [ object Function ] ",xt=" [ object GeneratorFunction ] ",Le=" [ object Map ] ",ut=" [ object Number ] ",Rt=" [ object Null ] ",at=" [ object Object ] ",Ct=" [ object Promise ] ",ln=" [ object Proxy ] ",ue=" [ object RegExp ] ",Ie=" [ object Set ] ",We=" [ object String ] ",Xe=" [ object Symbol ] ",Ce=" [ object Undefined ] ",Ne=" [ object WeakMap ] ",kt=" [ object WeakSet ] ",Nt=" [ object ArrayBuffer ] ",Te=" [ object DataView ] ",Et=" [ object Float32Array ] ",Ut=" [ object Float64Array ] ",Ve=" [ object Int8Array ] ",ft=" [ object Int16Array ] ",un=" [ object Int32Array ] ",je=" [ object Uint8Array ] ",St=" [ object Uint8ClampedArray ] ",ct=" [ object Uint16Array ] ",Vt=" [ object Uint32Array ] ",hn=/\b__p \+= '';/g,Lr=/\b(__p \+=) '' \+/g,hr=/(__e\(.*?\)|\b__t\)) \+\n'';/g,Zr=/&(?:amp|lt|gt|quot|#39);/g,oa=/[&<>" ']/g,O1=RegExp(Zr.source),ja=RegExp(oa.source),kp=/<%-([\s\S]+?)%>/g,M1=/<%([\s\S]+?)%>/g,wc=/<%=([\s\S]+?)%>/g,Ep=/\.|\[(?:[^[\]]*|(["' ] ) ( ? : ( ? ! \ 1 ) [ ^ \ \ ] | \ \ . ) * ? \ 1 ) \ ] / , Pp = /^\w*$/ , aa = /[^.[\]]+|\[(?:(-?\d+(?:\.\d+)?)|(["'])((?:(?!\2)[^\\]|\\.)*?)\2)\]|(?=(?:\.|\[\])(?:\.|\[\]|$))/g , wf = /[\\^$.*+?()[\]{}|]/g , R1 = RegExp ( wf . source ) , Cc = /^\s+/ , Cf = /\s/ , I1 = /\{(?:\n\/\* \[wrapped with .+\] \*\/)?\n?/ , sl = /\{\n\/\* \[wrapped with (.+)\] \*/ , _c = /,? & / , D1 = /[^\x00-\x2f\x3a-\x40\x5b-\x60\x7b-\x7f]+/g , N1 = /[()=,{}\[\]\/\s]/ , B1 = /\\(\\)?/g , $1 = /\$\{([^\\}]*(?:\\.[^\\}]*)*)\}/g , Ss = /\w*$/ , F1 = /^[-+]0x[0-9a-f]+$/i , z1 = /^0b[01]+$/i , j1 = /^\[object .+?Constructor\]$/ , H1 = /^0o[0-7]+$/i , V1 = /^(?:0|[1-9]\d*)$/ , U1 = /[\xc0-\xd6\xd8-\xf6\xf8-\xff\u0100-\u017f]/g , ll = /($^)/ , W1 = /['\n\r\u2028\u2029\\]/g , xs = "\\ud800-\\udfff" , xu = "\\u0300-\\u036f" , wu = "\\ufe20-\\ufe2f" , ul = "\\u20d0-\\u20ff" , Cu = xu + wu + ul , Tp = "\\u2700-\\u27bf" , kc = "a-z\\xdf-\\xf6\\xf8-\\xff" , cl = "\\xac\\xb1\\xd7\\xf7" , sa = "\\x00-\\x2f\\x3a-\\x40\\x5b-\\x60\\x7b-\\xbf" , Mn = "\\u2000-\\u206f" , kn = " \\t\\x0b\\f\\xa0\\ufeff\\n\\r\\u2028\\u2029\\u1680\\u180e\\u2000\\u2001\\u2002\\u2003\\u2004\\u2005\\u2006\\u2007\\u2008\\u2009\\u200a\\u202f\\u205f\\u3000" , la = "A-Z\\xc0-\\xd6\\xd8-\\xde" , zr = "\\ufe0e\\ufe0f" , si = cl + sa + Mn + kn , ua = "['\u2019]" , dl = "[" + xs + "]" , li = "[" + si + "]" , ws = "[" + Cu + "]" , _f = "\\d+" , _u = "[" + Tp + "]" , Cs = "[" + kc + "]" , kf = "[^" + xs + si + _f + Tp + kc + la + "]" , Oi = "\\ud83c[\\udffb-\\udfff]" , Lp = "(?:" + ws + "|" + Oi + ")" , Ap = "[^" + xs + "]" , Ef = "(?:\\ud83c[\\udde6-\\uddff]){2}" , fl = "[\\ud800-\\udbff][\\udc00-\\udfff]" , Io = "[" + la + "]" , hl = "\\u200d" , ku = "(?:" + Cs + "|" + kf + ")" , G1 = "(?:" + Io + "|" + kf + ")" , Ec = "(?:" + ua + "(?:d|ll|m|re|s|t|ve))?" , Pc = "(?:" + ua + "(?:D|LL|M|RE|S|T|VE))?" , Pf = Lp + "?" , Tc = "[" + zr + "]?" , Ha = "(?:" + hl + "(?:" + [ Ap , Ef , fl ] . join ( "|" ) + ")" + Tc + Pf + ")*" , Tf = "\\d*(?:1st|2nd|3rd|(?![123])\\dth)(?=\\b|[A-Z_])" , Eu = "\\d*(?:1ST|2ND|3RD|(?![123])\\dTH)(?=\\b|[a-z_])" , Xt = Tc + Pf + Ha , Op = "(?:" + [ _u , Ef , fl ] . join ( "|" ) + ")" + Xt , Lc = "(?:" + [ Ap + ws + "?" , ws , Ef , fl , dl ] . join ( "|" ) + ")" , Ac = RegExp ( ua , "g" ) , Mp = RegExp ( ws , "g" ) , ca = RegExp ( Oi + "(?=" + Oi + ")|" + Lc + Xt , "g" ) , Kn = RegExp ( [ Io + "?" + Cs + "+" + Ec + "(?=" + [ li , Io , "$" ] . join ( "|" ) + ")" , G1 + "+" + Pc + "(?=" + [ li , Io + ku , "$" ] . join ( "|" ) + ")" , Io + "?" + ku + "+" + Ec , Io + "+" + Pc , Eu , Tf , _f , Op ] . join ( "|" ) , "g" ) , Lf = RegExp ( "[" + hl + xs + Cu + zr + "]" ) , Rp = /[a-z][A-Z]|[A-Z]{2}[a-z]|[0-9][a-zA-Z]|[a-zA-Z][0-9]|[^a-zA-Z0-9 ]/ , Af = [ "Array" , "Buffer" , "DataView" , "Date" , "Error" , "Float32Array" , "Float64Array" , "Function" , "Int8Array" , "Int16Array" , "Int32Array" , "Map" , "Math" , "Object" , "Promise" , "RegExp" , "Set" , "String" , "Symbol" , "TypeError" , "Uint8Array" , "Uint8ClampedArray" , "Uint16Array" , "Uint32Array" , "WeakMap" , "_" , "clearTimeout" , "isFinite" , "parseInt" , "setTimeout" ] , Ip = - 1 , mn = { } ; mn
2022-12-26 15:23:16 +00:00
/* [wrapped with `+g+`] */
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` )}function EK(c){return jt(c)||sh(c)||!!(im&&c&&c[im])}function Nu(c,g){var w=typeof c;return g=g==null?U:g,!!g&&(w=="number"||w!="symbol"&&V1.test(c))&&c>-1&&c%1==0&&c<g}function yo(c,g,w){if(!vr(w))return!1;var O=typeof g;return(O=="number"?Vo(w)&&Nu(g,w.length):O=="string"&&g in w)?Rs(w[g],c):!1}function bw(c,g){if(jt(c))return!1;var w=typeof c;return w=="number"||w=="symbol"||w=="boolean"||c==null||xa(c)?!0:Pp.test(c)||!Ep.test(c)||g!=null&&c in vn(g)}function PK(c){var g=typeof c;return g=="string"||g=="number"||g=="symbol"||g=="boolean"?c!=="__proto__":c===null}function Sw(c){var g=xe(c),w=F[g];if(typeof w!="function"||!(g in Jt.prototype))return!1;if(c===w)return!0;var O=ce(w);return!!O&&c===O[0]}function TK(c){return!! $ c&& $ c in c}var LK=ur?Bu:Bw;function Dm(c){var g=c&&c.constructor,w=typeof g=="function"&&g.prototype||En;return c===w}function lT(c){return c===c&&!vr(c)}function uT(c,g){return function(w){return w==null?!1:w[c]===g&&(g!==n||c in vn(w))}}function AK(c){var g=C3(c,function(O){return w.size===u&&w.clear(),O}),w=g.cache;return g}function OK(c,g){var w=c[1],O=g[1],N=w|O,H=N<(E|k|D),Q=O==D&&w==T||O==D&&w==B&&c[7].length<=g[8]||O==(D|B)&&g[7].length<=g[8]&&w==T;if(!(H||Q))return c;O&E&&(c[2]=g[2],N|=w&E?0:_);var ne=g[3];if(ne){var de=c[3];c[3]=de?v3(de,ne,g[4]):ne,c[4]=de?ha(c[3],f):g[4]}return ne=g[5],ne&&(de=c[5],c[5]=de?Am(de,ne,g[6]):ne,c[6]=de?ha(c[5],f):g[6]),ne=g[7],ne&&(c[7]=ne),O&D&&(c[8]=c[8]==null?g[8]:di(c[8],g[8])),c[9]==null&&(c[9]=g[9]),c[0]=g[0],c[1]=N,c}function MK(c){var g=[];if(c!=null)for(var w in vn(c))g.push(w);return g}function RK(c){return Hp.call(c)}function cT(c,g,w){return g=Or(g===n?c.length-1:g,0),function(){for(var O=arguments,N=-1,H=Or(O.length-g,0),Q=ye(H);++N<H;)Q[N]=O[g+N];N=-1;for(var ne=ye(g+1);++N<g;)ne[N]=O[N];return ne[g]=w(Q),Mi(c,this,ne)}}function dT(c,g){return g.length<2?c:Sl(c,Wr(g,0,-1))}function IK(c,g){for(var w=c.length,O=di(g.length,w),N= $ i(c);O--;){var H=g[O];c[O]=Nu(H,w)?N[H]:n}return c}function xw(c,g){if(!(g==="constructor"&&typeof c[g]=="function")&&g!="__proto__")return c[g]}var fT=pT(eh),Nm=Uf||function(c,g){return Pt.setTimeout(c,g)},ww=pT(u0);function hT(c,g,w){var O=g+"";return ww(c,kK(O,DK(wK(O),w)))}function pT(c){var g=0,w=0;return function(){var O=om(),N=G-(O-w);if(w=O,N>0){if(++g>=te)return arguments[0]}else g=0;return c.apply(n,arguments)}}function b3(c,g){var w=-1,O=c.length,N=O-1;for(g=g===n?O:g;++w<g;){var H=Qf(w,N),Q=c[H];c[H]=c[w],c[w]=Q}return c.length=g,c}var gT=AK(function(c){var g=[];return c.charCodeAt(0)===46&&g.push(""),c.replace(aa,function(w,O,N,H){g.push(N?H.replace(B1," $ 1"):O||w)}),g});function Al(c){if(typeof c=="string"||xa(c))return c;var g=c+"";return g=="0"&&1/c==-Z?"-0":g}function ah(c){if(c!=null){try{return pr.call(c)}catch(g){}try{return c+""}catch(g){}}return""}function DK(c,g){return Xn(be,function(w){var O="_."+w[0];g&w[1]&&!Zi(c,O)&&c.push(O)}),c.sort()}function mT(c){if(c instanceof Jt)return c.clone();var g=new po(c.__wrapped__,c.__chain__);return g.__actions__= $ i(c.__actions__),g.__index__=c.__index__,g.__values__=c.__values__,g}function NK(c,g,w){(w?yo(c,g,w):g===n)?g=1:g=Or(Wt(g),0);var O=c==null?0:c.length;if(!O||g<1)return[];for(var N=0,H=0,Q=ye(Wf(O/g));N<O;)Q[H++]=Wr(c,N,N+=g);return Q}function BK(c){for(var g=-1,w=c==null?0:c.length,O=0,N=[];++g<w;){var H=c[g];H&&(N[O++]=H)}return N}function $ K(){var c=arguments.length;if(!c)return[];for(var g=ye(c-1),w=arguments[0],O=c;O--;)g[O-1]=arguments[O];return Ua(jt(w)? $ i(w):[w],Vr(g,1))}var FK=Mt(function(c,g){return Gr(c)?Qc(c,Vr(g,1,Gr,!0)):[]}),zK=Mt(function(c,g){var w=Ka(g);return Gr(w)&&(w=n),Gr(c)?Qc(c,Vr(g,1,Gr,!0),Me(w,2)):[]}),jK=Mt(function(c,g){var w=Ka(g);return Gr(w)&&(w=n),Gr(c)?Qc(c,Vr(g,1,Gr,!0),n,w):[]});function HK(c,g,w){var O=c==null?0:c.length;return O?(g=w||g===n?1:Wt(g),Wr(c,g<0?0:g,O)):[]}function VK(c,g,w){var O=c==null?0:c.length;return O?(g=w||g===n?1:Wt(g),g=O-g,Wr(c,0,g<0?0:g)):[]}function UK(c,g){return c&&c.length?od(c,Me(g,3),!0,!0):[]}function WK(c,g){return c&&c.length?od(c,Me(g,3),!0):[]}function GK(c,g,w,O){var N=c==nul
` ;c.replace(nt,function(At,en,an,wa,bo,Ca){return an||(an=wa),Re+=c.slice(Ee,Ca).replace(W1,ml),en&&(ne=!0,Re+= ` ' +
2022-12-26 15:23:16 +00:00
_ _e ( ` +en+ ` ) +
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
'`),bo&&(de=!0,Re+=`' ;
` +bo+ ` ;
2022-12-26 15:23:16 +00:00
_ _p += '`),an&&(Re+=`' +
( ( _ _t = ( ` +an+ ` ) ) == null ? '' : _ _t ) +
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
'`),Ee=Ca+At.length,At}),Re+=`' ;
2022-12-26 15:23:16 +00:00
` ;var Lt=cn.call(g,"variable")&&g.variable;if(!Lt)Re= ` with ( obj ) {
` +Re+ `
}
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` ;else if(N1.test(Lt))throw new It(s);Re=(de?Re.replace(hn,""):Re).replace(Lr," $ 1").replace(hr," $ 1;"),Re="function("+(Lt||"obj")+ ` ) {
2022-12-26 15:23:16 +00:00
` +(Lt?"": ` obj || ( obj = { } ) ;
` )+"var __t, __p = ''"+(ne?", __e = _.escape":"")+(de? ` , _ _j = Array . prototype . join ;
function print ( ) { _ _p += _ _j . call ( arguments , '' ) }
` : ` ;
` )+Re+ ` return _ _p
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
} ` ;var Kt=YT(function(){return on(H,wt+"return "+Re).apply(n,Q)});if(Kt.source=Re,Pw(Kt))throw Kt;return Kt}function tJ(c){return Pn(c).toLowerCase()}function nJ(c){return Pn(c).toUpperCase()}function rJ(c,g,w){if(c=Pn(c),c&&(w||g===n))return ho(c);if(!c||!(g=mo(g)))return c;var O=Qi(c),N=Qi(g),H=fa(O,N),Q=_s(O,N)+1;return Os(O,H,Q).join("")}function iJ(c,g,w){if(c=Pn(c),c&&(w||g===n))return c.slice(0,J1(c)+1);if(!c||!(g=mo(g)))return c;var O=Qi(c),N=_s(O,Qi(g))+1;return Os(O,0,N).join("")}function oJ(c,g,w){if(c=Pn(c),c&&(w||g===n))return c.replace(Cc,"");if(!c||!(g=mo(g)))return c;var O=Qi(c),N=fa(O,Qi(g));return Os(O,N).join("")}function aJ(c,g){var w=j,O=K;if(vr(g)){var N="separator"in g?g.separator:N;w="length"in g?Wt(g.length):w,O="omission"in g?mo(g.omission):O}c=Pn(c);var H=c.length;if(Tu(c)){var Q=Qi(c);H=Q.length}if(w>=H)return c;var ne=w-Wa(O);if(ne<1)return O;var de=Q?Os(Q,0,ne).join(""):c.slice(0,ne);if(N===n)return de+O;if(Q&&(ne+=de.length-ne),Tw(N)){if(c.slice(ne).search(N)){var Ee,Pe=de;for(N.global||(N=zf(N.source,Pn(Ss.exec(N))+"g")),N.lastIndex=0;Ee=N.exec(Pe);)var Re=Ee.index;de=de.slice(0,Re===n?ne:Re)}}else if(c.indexOf(mo(N),ne)!=ne){var nt=de.lastIndexOf(N);nt>-1&&(de=de.slice(0,nt))}return de+O}function sJ(c){return c=Pn(c),c&&O1.test(c)?c.replace(Zr,Ky):c}var lJ=Ll(function(c,g,w){return c+(w?" ":"")+g.toUpperCase()}),Ow=v0("toUpperCase");function qT(c,g,w){return c=Pn(c),g=w?n:g,g===n?Fp(c)?Ff(c):X1(c):c.match(g)||[]}var YT=Mt(function(c,g){try{return Mi(c,n,g)}catch(w){return Pw(w)?w:new It(w)}}),uJ=cr(function(c,g){return Xn(g,function(w){w=Al(w),ma(c,w,kw(c[w],c))}),c});function cJ(c){var g=c==null?0:c.length,w=Me();return c=g?Vn(c,function(O){if(typeof O[1]!="function")throw new Ri(a);return[w(O[0]),O[1]]}):[],Mt(function(O){for(var N=-1;++N<g;){var H=c[N];if(Mi(H[0],this,O))return Mi(H[1],this,O)}})}function dJ(c){return Kp(wi(c,p))}function Mw(c){return function(){return c}}function fJ(c,g){return c==null||c!==c?g:c}var hJ=Rm(),pJ=Rm(!0);function Wo(c){return c}function Rw(c){return mr(typeof c=="function"?c:wi(c,p))}function gJ(c){return nd(wi(c,p))}function mJ(c,g){return wl(c,wi(g,p))}var vJ=Mt(function(c,g){return function(w){return Ls(w,c,g)}}),yJ=Mt(function(c,g){return function(w){return Ls(c,w,g)}});function Iw(c,g,w){var O=Ei(g),N=Jp(g,O);w==null&&!(vr(g)&&(N.length||!O.length))&&(w=g,g=c,c=this,N=Jp(g,Ei(g)));var H=!(vr(w)&&"chain"in w)||!!w.chain,Q=Bu(c);return Xn(N,function(ne){var de=g[ne];c[ne]=de,Q&&(c.prototype[ne]=function(){var Ee=this.__chain__;if(H||Ee){var Pe=c(this.__wrapped__),Re=Pe.__actions__= $ i(this.__actions__);return Re.push({func:de,args:arguments,thisArg:c}),Pe.__chain__=Ee,Pe}return de.apply(c,Ua([this.value()],arguments))})}),c}function bJ(){return Pt._===this&&(Pt._=tm),this}function Dw(){}function SJ(c){return c=Wt(c),Mt(function(g){return rd(g,c)})}var xJ=Im(Vn),wJ=Im(Hn),CJ=Im(Oc);function KT(c){return bw(c)?gl(Al(c)):As(c)}function _J(c){return function(g){return c==null?n:Sl(c,g)}}var kJ=S(),EJ=S(!0);function Nw(){return[]}function Bw(){return!1}function PJ(){return{}}function TJ(){return""}function LJ(){return!0}function AJ(c,g){if(c=Wt(c),c<1||c>U)return[];var w=fe,O=di(c,fe);g=Me(g),c-=fe;for(var N=Nf(O,g);++w<c;)g(w);return N}function OJ(c){return jt(c)?Vn(c,Al):xa(c)?[c]: $ i(gT(Pn(c)))}function MJ(c){var g=++jf;return Pn(c)+g}var RJ=y0(function(c,g){return c+g},0),IJ=M("ceil"),DJ=y0(function(c,g){return c/g},1),NJ=M("floor");function BJ(c){return c&&c.length?Rr(c,Wo,e0):n}function $ J(c,g){return c&&c.length?Rr(c,Me(g,2),e0):n}function FJ(c){return Rf(c,Wo)}function zJ(c,g){return Rf(c,Me(g,2))}function jJ(c){return c&&c.length?Rr(c,Wo,xl):n}function HJ(c,g){return c&&c.length?Rr(c,Me(g,2),xl):n}var VJ=y0(function(c,g){return c*g},1),UJ=M("round"),WJ=y0(function(c,g){return c-g},0);function GJ(c){return c&&c.length?Df(c,Wo):0}function qJ(c,g){return c&&c.length?Df(c,Me(g,2)):0}return F.after=gZ,F.ary=ET,F.assign=nQ,F.assignIn=zT,F.assignInWith=P3,F.assignWith=rQ,F.at=iQ,F.before=PT,F.bind=kw,F.bindAll=uJ,F.bindKey=TT,F.castArray=PZ,F.chain=CT,F.c
In order to be iterable , non - array objects must have a [ Symbol . iterator ] ( ) method . ` )}function TSe(e){return _Se(e)||kSe(e)||ESe(e)||PSe()}function ER(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter(function(i){return Object.getOwnPropertyDescriptor(e,i).enumerable})),n.push.apply(n,r)}return n}function PR(e){for(var t=1;t<arguments.length;t++){var n=arguments[t]!=null?arguments[t]:{};t%2?ER(Object(n),!0).forEach(function(r){pf(e,r,n[r])}):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):ER(Object(n)).forEach(function(r){Object.defineProperty(e,r,Object.getOwnPropertyDescriptor(n,r))})}return e}var LSe={type:"logger",log:function(t){this.output("log",t)},warn:function(t){this.output("warn",t)},error:function(t){this.output("error",t)},output:function(t,n){console&&console[t]&&console[t].apply(console,n)}},ASe=function(){function e(t){var n=arguments.length>1&&arguments[1]!==void 0?arguments[1]:{};vu(this,e),this.init(t,n)}return yu(e,[{key:"init",value:function(n){var r=arguments.length>1&&arguments[1]!==void 0?arguments[1]:{};this.prefix=r.prefix||"i18next:",this.logger=n||LSe,this.options=r,this.debug=r.debug}},{key:"setDebug",value:function(n){this.debug=n}},{key:"log",value:function(){for(var n=arguments.length,r=new Array(n),i=0;i<n;i++)r[i]=arguments[i];return this.forward(r,"log","",!0)}},{key:"warn",value:function(){for(var n=arguments.length,r=new Array(n),i=0;i<n;i++)r[i]=arguments[i];return this.forward(r,"warn","",!0)}},{key:"error",value:function(){for(var n=arguments.length,r=new Array(n),i=0;i<n;i++)r[i]=arguments[i];return this.forward(r,"error","")}},{key:"deprecate",value:function(){for(var n=arguments.length,r=new Array(n),i=0;i<n;i++)r[i]=arguments[i];return this.forward(r,"warn","WARNING DEPRECATED: ",!0)}},{key:"forward",value:function(n,r,i,o){return o&&!this.debug?null:(typeof n[0]=="string"&&(n[0]="".concat(i).concat(this.prefix," ").concat(n[0])),this.logger[r](n))}},{key:"create",value:function(n){return new e(this.logger,PR(PR({},{prefix:"".concat(this.prefix,":").concat(n,":")}),this.options))}},{key:"clone",value:function(n){return n=n||this.options,n.prefix=n.prefix||this.prefix,new e(this.logger,n)}}]),e}(),Kl=new ASe,tf=function(){function e(){vu(this,e),this.observers={}}return yu(e,[{key:"on",value:function(n,r){var i=this;return n.split(" ").forEach(function(o){i.observers[o]=i.observers[o]||[],i.observers[o].push(r)}),this}},{key:"off",value:function(n,r){if(!!this.observers[n]){if(!r){delete this.observers[n];return}this.observers[n]=this.observers[n].filter(function(i){return i!==r})}}},{key:"emit",value:function(n){for(var r=arguments.length,i=new Array(r>1?r-1:0),o=1;o<r;o++)i[o-1]=arguments[o];if(this.observers[n]){var a=[].concat(this.observers[n]);a.forEach(function(l){l.apply(void 0,i)})}if(this.observers["*"]){var s=[].concat(this.observers["*"]);s.forEach(function(l){l.apply(l,[n].concat(i))})}}}]),e}();function tv(){var e,t,n=new Promise(function(r,i){e=r,t=i});return n.resolve=e,n.reject=t,n}function TR(e){return e==null?"":""+e}function OSe(e,t,n){e.forEach(function(r){t[r]&&(n[r]=t[r])})}function qE(e,t,n){function r(s){return s&&s.indexOf("###")>-1?s.replace(/###/g,"."):s}function i(){return!e||typeof e=="string"}for(var o=typeof t!="string"?[].concat(t):t.split(".");o.length>1;){if(i())return{};var a=r(o.shift());!e[a]&&n&&(e[a]=new n),Object.prototype.hasOwnProperty.call(e,a)?e=e[a]:e={}}return i()?{}:{obj:e,k:r(o.shift())}}function LR(e,t,n){var r=qE(e,t,Object),i=r.obj,o=r.k;i[o]=n}function MSe(e,t,n,r){var i=qE(e,t,Object),o=i.obj,a=i.k;o[a]=o[a]||[],r&&(o[a]=o[a].concat(n)),r||o[a].push(n)}function B5(e,t){var n=qE(e,t),r=n.obj,i=n.k;if(!!r)return r[i]}function AR(e,t,n){var r=B5(e,n);return r!==void 0?r:B5(t,n)}function zU(e,t,n){for(var r in t)r!=="__proto__"&&r!=="constructor"&&(r in e?typeof e[r]=="string"||e[r]instanceof String||typeof t[r]=="string"||t[r]instanceof String?n&&(e[r]=t[r]):zU(e[r],t[r],n):e[r]=t[r]);return e}function A0(e){return e.rep
In order to be iterable , non - array objects must have a [ Symbol . iterator ] ( ) method . ` )}function twe(e,t){return Zxe(e)||Qxe(e,t)||Jxe(e,t)||ewe()}function aI(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter(function(i){return Object.getOwnPropertyDescriptor(e,i).enumerable})),n.push.apply(n,r)}return n}function fC(e){for(var t=1;t<arguments.length;t++){var n=arguments[t]!=null?arguments[t]:{};t%2?aI(Object(n),!0).forEach(function(r){ZU(e,r,n[r])}):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):aI(Object(n)).forEach(function(r){Object.defineProperty(e,r,Object.getOwnPropertyDescriptor(n,r))})}return e}var nwe=function(t,n){var r=C.exports.useRef();return C.exports.useEffect(function(){r.current=n?r.current:t},[t,n]),r.current};function Ye(e){var t=arguments.length>1&&arguments[1]!==void 0?arguments[1]:{},n=t.i18n,r=C.exports.useContext(Kxe)||{},i=r.i18n,o=r.defaultNS,a=n||i||Wxe();if(a&&!a.reportNamespaces&&(a.reportNamespaces=new Xxe),!a){i8("You will need to pass in an i18next instance by using initReactI18next");var s=function(z){return Array.isArray(z)?z[z.length-1]:z},l=[s,{},!1];return l.t=s,l.i18n={},l.ready=!1,l}a.options.react&&a.options.react.wait!==void 0&&i8("It seems you are still using the old wait option, you may migrate to the new useSuspense behaviour.");var u=fC(fC(fC({},Vxe()),a.options.react),t),f=u.useSuspense,p=u.keyPrefix,m=e||o||a.options&&a.options.defaultNS;m=typeof m=="string"?[m]:m||["translation"],a.reportNamespaces.addUsedNamespaces&&a.reportNamespaces.addUsedNamespaces(m);var v=(a.isInitialized||a.initializedStoreOnce)&&m.every(function(B){return Bxe(B,a,u)});function b(){return a.getFixedT(null,u.nsMode==="fallback"?m:m[0],p)}var x=C.exports.useState(b),E=twe(x,2),k=E[0],_=E[1],T=m.join(),A=nwe(T),R=C.exports.useRef(!0);C.exports.useEffect(function(){var B=u.bindI18n,z=u.bindI18nStore;R.current=!0,!v&&!f&&tI(a,m,function(){R.current&&_(b)}),v&&A&&A!==T&&R.current&&_(b);function j(){R.current&&_(b)}return B&&a&&a.on(B,j),z&&a&&a.store.on(z,j),function(){R.current=!1,B&&a&&B.split(" ").forEach(function(K){return a.off(K,j)}),z&&a&&z.split(" ").forEach(function(K){return a.store.off(K,j)})}},[a,T]);var I=C.exports.useRef(!0);C.exports.useEffect(function(){R.current&&!I.current&&_(b),I.current=!1},[a,p]);var D=[k,a,v];if(D.t=k,D.i18n=a,D.ready=v,v||!v&&!f)return D;throw new Promise(function(B){tI(a,m,function(){B()})})}Ht.use(KU).use(UU).use(Yxe).init({fallbackLng:"en",debug:!1,ns:["common","gallery","hotkeys","options","settings","modelmanager","toast","tooltip","unifiedcanvas"],backend:{loadPath:"/locales/{{ns}}/{{lng}}.json"},interpolation:{escapeValue:!1},returnNull:!1});const rwe={isConnected:!1,isProcessing:!1,log:[],shouldShowLogViewer:!1,shouldDisplayInProgressType:"latents",shouldDisplayGuides:!0,isGFPGANAvailable:!0,isESRGANAvailable:!0,socketId:"",shouldConfirmOnDelete:!0,openAccordions:[],currentStep:0,totalSteps:0,currentIteration:0,totalIterations:0,currentStatus:Ht.isInitialized?Ht.t("common:statusDisconnected"):"Disconnected",currentStatusHasSteps:!1,model:"",model_id:"",model_hash:"",app_id:"",app_version:"",model_list:{},infill_methods:[],hasError:!1,wasErrorSeen:!0,isCancelable:!0,saveIntermediatesInterval:5,enableImageDebugging:!1,toastQueue:[],searchFolder:null,foundModels:null,openModel:null},JU=gx({name:"system",initialState:rwe,reducers:{setShouldDisplayInProgressType:(e,t)=>{e.shouldDisplayInProgressType=t.payload},setIsProcessing:(e,t)=>{e.isProcessing=t.payload},setCurrentStatus:(e,t)=>{e.currentStatus=t.payload},setSystemStatus:(e,t)=>({...e,...t.payload}),errorOccurred:e=>{e.hasError=!0,e.isProcessing=!1,e.isCancelable=!0,e.currentStep=0,e.totalSteps=0,e.currentIteration=0,e.totalIterations=0,e.currentStatusHasSteps=!1,e.currentStatus=Ht.t("common:statusError"),e.wasErrorSeen=!1},errorSeen:e=>{e.hasError=!1,e.wasErrorSeen=!0,e.currentStatus=e.isConnected?Ht.t("common:statusConnected"):Ht.t("common:statusDisconnected")},addLogEntry:(e,t)=>{const{timestamp:n,
` ),n=+this.fontSize(),r=0,i=this.lineHeight()*n,o=this.attrs.width,a=this.attrs.height,s=o!==D0&&o!==void 0,l=a!==D0&&a!==void 0,u=this.padding(),f=o-u*2,p=a-u*2,m=0,v=this.wrap(),b=v!==WI,x=v!==k7e&&b,E=this.ellipsis();this.textArr=[],_C().font=this._getContextFont();for(var k=E?this._getTextWidth(CC):0,_=0,T=t.length;_<T;++_){var A=t[_],R=this._getTextWidth(A);if(s&&R>f)for(;A.length>0;){for(var I=0,D=A.length,B="",z=0;I<D;){var j=I+D>>>1,K=A.slice(0,j+1),te=this._getTextWidth(K)+k;te<=f?(I=j+1,B=K,z=te):D=j}if(B){if(x){var G, $ =A[B.length],V= $ ===Rb|| $ ===VI;V&&z<=f?G=B.length:G=Math.max(B.lastIndexOf(Rb),B.lastIndexOf(VI))+1,G>0&&(I=G,B=B.slice(0,I),z=this._getTextWidth(B))}B=B.trimRight(),this._addTextLine(B),r=Math.max(r,z),m+=i;var J=this._shouldHandleEllipsis(m);if(J){this._tryToAddEllipsisToLastLine();break}if(A=A.slice(I),A=A.trimLeft(),A.length>0&&(R=this._getTextWidth(A),R<=f)){this._addTextLine(A),m+=i,r=Math.max(r,R);break}}else break}else this._addTextLine(A),m+=i,r=Math.max(r,R),this._shouldHandleEllipsis(m)&&_<T-1&&this._tryToAddEllipsisToLastLine();if(l&&m+i>p)break;this.textArr[this.textArr.length-1]&&(this.textArr[this.textArr.length-1].lastInParagraph=!0)}this.textHeight=n,this.textWidth=r}_shouldHandleEllipsis(t){var n=+this.fontSize(),r=this.lineHeight()*n,i=this.attrs.height,o=i!==D0&&i!==void 0,a=this.padding(),s=i-a*2,l=this.wrap(),u=l!==WI;return!u||o&&t+r>s}_tryToAddEllipsisToLastLine(){var t=this.attrs.width,n=t!==D0&&t!==void 0,r=this.padding(),i=t-r*2,o=this.ellipsis(),a=this.textArr[this.textArr.length-1];if(!(!a||!o)){if(n){var s=this._getTextWidth(a.text+CC)<i;s||(a.text=a.text.slice(0,a.text.length-3))}this.textArr.splice(this.textArr.length-1,1),this._addTextLine(a.text+CC)}}getStrokeScaleEnabled(){return!0}}kr.prototype._fillFunc=T7e;kr.prototype._strokeFunc=L7e;kr.prototype.className=b7e;kr.prototype._attrsAffectingSize=["text","fontSize","padding","wrap","lineHeight","letterSpacing"];Er(kr);ee.overWriteSetter(kr,"width",tP());ee.overWriteSetter(kr,"height",tP());ee.addGetterSetter(kr,"fontFamily","Arial");ee.addGetterSetter(kr,"fontSize",12,Ge());ee.addGetterSetter(kr,"fontStyle",cG);ee.addGetterSetter(kr,"fontVariant",cG);ee.addGetterSetter(kr,"padding",0,Ge());ee.addGetterSetter(kr,"align",uG);ee.addGetterSetter(kr,"verticalAlign",S7e);ee.addGetterSetter(kr,"lineHeight",1,Ge());ee.addGetterSetter(kr,"wrap",_7e);ee.addGetterSetter(kr,"ellipsis",!1,ol());ee.addGetterSetter(kr,"letterSpacing",0,Ge());ee.addGetterSetter(kr,"text","",k1());ee.addGetterSetter(kr,"textDecoration","");var O7e="",fG="normal";function hG(e){e.fillText(this.partialText,0,0)}function pG(e){e.strokeText(this.partialText,0,0)}class bi extends $ e{constructor(t){super(t),this.dummyCanvas=he.createCanvasElement(),this.dataArray=[],this.dataArray=zn.parsePathData(this.attrs.data),this.on("dataChange.konva",function(){this.dataArray=zn.parsePathData(this.attrs.data),this._setTextData()}),this.on("textChange.konva alignChange.konva letterSpacingChange.konva kerningFuncChange.konva fontSizeChange.konva fontFamilyChange.konva",this._setTextData),this._setTextData()}_sceneFunc(t){t.setAttr("font",this._getContextFont()),t.setAttr("textBaseline",this.textBaseline()),t.setAttr("textAlign","left"),t.save();var n=this.textDecoration(),r=this.fill(),i=this.fontSize(),o=this.glyphInfo;n==="underline"&&t.beginPath();for(var a=0;a<o.length;a++){t.save();var s=o[a].p0;t.translate(s.x,s.y),t.rotate(o[a].rotation),this.partialText=o[a].text,t.fillStrokeShape(this),n==="underline"&&(a===0&&t.moveTo(0,i/2+1),t.lineTo(i,i/2+1)),t.restore()}n==="underline"&&(t.strokeStyle=r,t.lineWidth=i/20,t.stroke()),t.restore()}_hitFunc(t){t.beginPath();var n=this.glyphInfo;if(n.length>=1){var r=n[0].p0;t.moveTo(r.x,r.y)}for(var i=0;i<n.length;i++){var o=n[i].p1;t.lineTo(o.x,o.y)}t.setAttr("lineWidth",this.fontSize()),t.setAttr("strokeStyle",this.colorKey),t.stroke()}getTextWidth(){return this.textWidth}getTextHeight(){return he.warn("text.getTextHeight() method is deprecated. Use text.height() - for full height and text.fontSize() - for one line hei
2022-12-26 15:23:16 +00:00
* @ license React
* use - sync - external - store - shim . production . min . js
*
* Copyright ( c ) Facebook , Inc . and its affiliates .
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree .
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
* / v a r r 1 = C . e x p o r t s ; f u n c t i o n L 9 e ( e , t ) { r e t u r n e = = = t & & ( e ! = = 0 | | 1 / e = = = 1 / t ) | | e ! = = e & & t ! = = t } v a r A 9 e = t y p e o f O b j e c t . i s = = " f u n c t i o n " ? O b j e c t . i s : L 9 e , O 9 e = r 1 . u s e S t a t e , M 9 e = r 1 . u s e E f f e c t , R 9 e = r 1 . u s e L a y o u t E f f e c t , I 9 e = r 1 . u s e D e b u g V a l u e ; f u n c t i o n D 9 e ( e , t ) { v a r n = t ( ) , r = O 9 e ( { i n s t : { v a l u e : n , g e t S n a p s h o t : t } } ) , i = r [ 0 ] . i n s t , o = r [ 1 ] ; r e t u r n R 9 e ( f u n c t i o n ( ) { i . v a l u e = n , i . g e t S n a p s h o t = t , E C ( i ) & & o ( { i n s t : i } ) } , [ e , n , t ] ) , M 9 e ( f u n c t i o n ( ) { r e t u r n E C ( i ) & & o ( { i n s t : i } ) , e ( f u n c t i o n ( ) { E C ( i ) & & o ( { i n s t : i } ) } ) } , [ e ] ) , I 9 e ( n ) , n } f u n c t i o n E C ( e ) { v a r t = e . g e t S n a p s h o t ; e = e . v a l u e ; t r y { v a r n = t ( ) ; r e t u r n ! A 9 e ( e , n ) } c a t c h ( r ) { r e t u r n ! 0 } } f u n c t i o n N 9 e ( e , t ) { r e t u r n t ( ) } v a r B 9 e = t y p e o f w i n d o w > " u " | | t y p e o f w i n d o w . d o c u m e n t > " u " | | t y p e o f w i n d o w . d o c u m e n t . c r e a t e E l e m e n t > " u " ? N 9 e : D 9 e ; C G . u s e S y n c E x t e r n a l S t o r e = r 1 . u s e S y n c E x t e r n a l S t o r e ! = = v o i d 0 ? r 1 . u s e S y n c E x t e r n a l S t o r e : B 9 e ; ( f u n c t i o n ( e ) { e . e x p o r t s = C G } ) ( w G ) ; v a r _ G = { e x p o r t s : { } } , k G = { } ; / * *
2022-12-26 15:23:16 +00:00
* @ license React
* use - sync - external - store - shim / with - selector . production . min . js
*
* Copyright ( c ) Facebook , Inc . and its affiliates .
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree .
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
* / v a r E x = C . e x p o r t s , $ 9 e = w G . e x p o r t s ; f u n c t i o n F 9 e ( e , t ) { r e t u r n e = = = t & & ( e ! = = 0 | | 1 / e = = = 1 / t ) | | e ! = = e & & t ! = = t } v a r z 9 e = t y p e o f O b j e c t . i s = = " f u n c t i o n " ? O b j e c t . i s : F 9 e , j 9 e = $ 9 e . u s e S y n c E x t e r n a l S t o r e , H 9 e = E x . u s e R e f , V 9 e = E x . u s e E f f e c t , U 9 e = E x . u s e M e m o , W 9 e = E x . u s e D e b u g V a l u e ; k G . u s e S y n c E x t e r n a l S t o r e W i t h S e l e c t o r = f u n c t i o n ( e , t , n , r , i ) { v a r o = H 9 e ( n u l l ) ; i f ( o . c u r r e n t = = = n u l l ) { v a r a = { h a s V a l u e : ! 1 , v a l u e : n u l l } ; o . c u r r e n t = a } e l s e a = o . c u r r e n t ; o = U 9 e ( f u n c t i o n ( ) { f u n c t i o n l ( v ) { i f ( ! u ) { i f ( u = ! 0 , f = v , v = r ( v ) , i ! = = v o i d 0 & & a . h a s V a l u e ) { v a r b = a . v a l u e ; i f ( i ( b , v ) ) r e t u r n p = b } r e t u r n p = v } i f ( b = p , z 9 e ( f , v ) ) r e t u r n b ; v a r x = r ( v ) ; r e t u r n i ! = = v o i d 0 & & i ( b , x ) ? b : ( f = v , p = x ) } v a r u = ! 1 , f , p , m = n = = = v o i d 0 ? n u l l : n ; r e t u r n [ f u n c t i o n ( ) { r e t u r n l ( t ( ) ) } , m = = = n u l l ? v o i d 0 : f u n c t i o n ( ) { r e t u r n l ( m ( ) ) } ] } , [ t , n , r , i ] ) ; v a r s = j 9 e ( e , o [ 0 ] , o [ 1 ] ) ; r e t u r n V 9 e ( f u n c t i o n ( ) { a . h a s V a l u e = ! 0 , a . v a l u e = s } , [ s ] ) , W 9 e ( s ) , s } ; ( f u n c t i o n ( e ) { e . e x p o r t s = k G } ) ( _ G ) ; f u n c t i o n G 9 e ( e ) { e ( ) } l e t E G = G 9 e ; c o n s t q 9 e = e = > E G = e , Y 9 e = ( ) = > E G , n f = C . e x p o r t s . c r e a t e C o n t e x t ( n u l l ) ; f u n c t i o n P G ( ) { r e t u r n C . e x p o r t s . u s e C o n t e x t ( n f ) } c o n s t K 9 e = ( ) = > { t h r o w n e w E r r o r ( " u S E S n o t i n i t i a l i z e d ! " ) } ; l e t T G = K 9 e ; c o n s t X 9 e = e = > { T G = e } , Z 9 e = ( e , t ) = > e = = = t ; f u n c t i o n Q 9 e ( e = n f ) { c o n s t t = e = = = n f ? P G : ( ) = > C . e x p o r t s . u s e C o n t e x t ( e ) ; r e t u r n f u n c t i o n ( r , i = Z 9 e ) { c o n s t { s t o r e : o , s u b s c r i p t i o n : a , g e t S e r v e r S t a t e : s } = t ( ) , l = T G ( a . a d d N e s t e d S u b , o . g e t S t a t e , s | | o . g e t S t a t e , r , i ) ; r e t u r n C . e x p o r t s . u s e D e b u g V a l u e ( l ) , l } } c o n s t J 9 e = Q 9 e ( ) ; v a r e 8 e = { e x p o r t s : { } } , B n = { } ; / * *
2022-12-26 15:23:16 +00:00
* @ license React
* react - is . production . min . js
*
* Copyright ( c ) Facebook , Inc . and its affiliates .
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree .
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
* / v a r a P = S y m b o l . f o r ( " r e a c t . e l e m e n t " ) , s P = S y m b o l . f o r ( " r e a c t . p o r t a l " ) , P x = S y m b o l . f o r ( " r e a c t . f r a g m e n t " ) , T x = S y m b o l . f o r ( " r e a c t . s t r i c t _ m o d e " ) , L x = S y m b o l . f o r ( " r e a c t . p r o f i l e r " ) , A x = S y m b o l . f o r ( " r e a c t . p r o v i d e r " ) , O x = S y m b o l . f o r ( " r e a c t . c o n t e x t " ) , t 8 e = S y m b o l . f o r ( " r e a c t . s e r v e r _ c o n t e x t " ) , M x = S y m b o l . f o r ( " r e a c t . f o r w a r d _ r e f " ) , R x = S y m b o l . f o r ( " r e a c t . s u s p e n s e " ) , I x = S y m b o l . f o r ( " r e a c t . s u s p e n s e _ l i s t " ) , D x = S y m b o l . f o r ( " r e a c t . m e m o " ) , N x = S y m b o l . f o r ( " r e a c t . l a z y " ) , n 8 e = S y m b o l . f o r ( " r e a c t . o f f s c r e e n " ) , L G ; L G = S y m b o l . f o r ( " r e a c t . m o d u l e . r e f e r e n c e " ) ; f u n c t i o n y s ( e ) { i f ( t y p e o f e = = " o b j e c t " & & e ! = = n u l l ) { v a r t = e . $ $ t y p e o f ; s w i t c h ( t ) { c a s e a P : s w i t c h ( e = e . t y p e , e ) { c a s e P x : c a s e L x : c a s e T x : c a s e R x : c a s e I x : r e t u r n e ; d e f a u l t : s w i t c h ( e = e & & e . $ $ t y p e o f , e ) { c a s e t 8 e : c a s e O x : c a s e M x : c a s e N x : c a s e D x : c a s e A x : r e t u r n e ; d e f a u l t : r e t u r n t } } c a s e s P : r e t u r n t } } } B n . C o n t e x t C o n s u m e r = O x ; B n . C o n t e x t P r o v i d e r = A x ; B n . E l e m e n t = a P ; B n . F o r w a r d R e f = M x ; B n . F r a g m e n t = P x ; B n . L a z y = N x ; B n . M e m o = D x ; B n . P o r t a l = s P ; B n . P r o f i l e r = L x ; B n . S t r i c t M o d e = T x ; B n . S u s p e n s e = R x ; B n . S u s p e n s e L i s t = I x ; B n . i s A s y n c M o d e = f u n c t i o n ( ) { r e t u r n ! 1 } ; B n . i s C o n c u r r e n t M o d e = f u n c t i o n ( ) { r e t u r n ! 1 } ; B n . i s C o n t e x t C o n s u m e r = f u n c t i o n ( e ) { r e t u r n y s ( e ) = = = O x } ; B n . i s C o n t e x t P r o v i d e r = f u n c t i o n ( e ) { r e t u r n y s ( e ) = = = A x } ; B n . i s E l e m e n t = f u n c t i o n ( e ) { r e t u r n t y p e o f e = = " o b j e c t " & & e ! = = n u l l & & e . $ $ t y p e o f = = = a P } ; B n . i s F o r w a r d R e f = f u n c t i o n ( e ) { r e t u r n y s ( e ) = = = M x } ; B n . i s F r a g m e n t = f u n c t i o n ( e ) { r e t u r n y s ( e ) = = = P x } ; B n . i s L a z y = f u n c t i o n ( e ) { r e t u r n y s ( e ) = = = N x } ; B n . i s M e m o = f u n c t i o n ( e ) { r e t u r n y s ( e ) = = = D x } ; B n . i s P o r t a l = f u n c t i o n ( e ) { r e t u r n y s ( e ) = = = s P } ; B n . i s P r o f i l e r = f u n c t i o n ( e ) { r e t u r n y s ( e ) = = = L x } ; B n . i s S t r i c t M o d e = f u n c t i o n ( e ) { r e t u r n y s ( e ) = = = T x } ; B n . i s S u s p e n s e = f u n c t i o n ( e ) { r e t u r n y s ( e ) = = = R x } ; B n . i s S u s p e n s e L i s t = f u n c t i o n ( e ) { r e t u r n y s ( e ) = = = I x } ; B n . i s V a l i d E l e m e n t T y p e = f u n c t i o n ( e ) { r e t u r n t y p e o f e = = " s t r i n g " | | t y p e o f e = = " f u n c t i o n " | | e = = = P x | | e = = = L x | | e = = = T x | | e = = = R x | | e = = = I x | | e = = = n 8 e | | t y p e o f e = = " o b j e c t " & & e ! = = n u l l & & ( e . $ $ t y p e o f = = = N x | | e . $ $ t y p e o f = = = D x | | e . $ $ t y p e o f = = = A x | | e . $ $ t y p e o f = = = O x | | e . $ $ t y p e o f = = = M x | | e . $ $ t y p e o f = = = L G | | e . g e t M o d u l e I d ! = = v o i d 0 ) } ; B n . t y p e O f = y s ; ( f u n c t i o n ( e ) { e . e x p o r t s = B n } ) ( e 8 e ) ; f u n c t i o n r 8 e ( ) { c o n s t e = Y 9 e ( ) ; l e t t = n u l l , n = n u l l ; r e t u r n { c l e a r ( ) { t = n u l l , n = n u l l } , n o t i f y ( ) { e ( ( ) = > { l e t r = t ; f o r ( ; r ; ) r . c a l l b a c k ( ) , r = r . n e x t } ) } , g e t ( ) { l e t r = [ ] , i = t ; f o r ( ; i ; ) r . p u s h ( i ) , i = i . n e x t ; r e t u r n r } , s u b s c r i b e ( r ) { l e t i = ! 0 , o = n = { c a l l b a c k : r , n e x t : n u l l , p r e v : n } ; r e t u r n o . p r e v ? o . p r e v . n e x t = o : t = o , f u n c t i o n ( ) { ! i | | t = = = n u l l | | ( i = ! 1 , o . n e x t ? o . n e x t . p r e v = o . p r e v : n = o . p r e v , o . p r e v ? o . p r e v . n e x t = o . n e x t : t = o . n e x t ) } } } } c o n s t K I = { n o t i f y ( ) { } , g e t : ( ) = > [ ] } ; f u n c t i o n i 8 e ( e , t ) { l e t n , r = K I ; f u n c t i o n i ( p ) { r e t u r n l ( ) , r . s u b s c r i b e ( p ) } f u n c t i o n o ( ) { r . n o t i f y ( ) } f u n c t i o n a ( ) { f . o n S t a t e C h a n g e & & f . o n S t a t e C h a n g e ( ) } f u n c t i o n s ( ) { r e t u r n B o o l e a n ( n ) } f u n c t i o n l ( ) { n | | ( n = t ? t . a d d N e s t e d S u b ( a ) : e . s u b s c r i b e ( a ) , r = r 8 e ( ) ) } f u n c t i o n u ( ) { n & & ( n ( ) , n = v o i d 0 , r . c l e a r ( ) , r = K I ) } c o n s t f = { a d d N e s t e d S u b : i , n o t i f y N e s t e d S u b s : o , h a n d l e C h a n g e W r a p p e r : a , i s S u b s c r i b e d : s , t r y S u b s c r i b e : l , t r y U n s u b s c r i b e : u , g e t L i s t e n e r s : ( ) = > r } ; r e t u r n f } c o n s t o 8 e = t y p e o f w i n d o w < " u " & & t y p e o f w i n d o w . d o c u m e n t < " u " & & t y p e o f w i n d o w . d o c u m e n t . c r e a t e E l e m e n t < " u " , a 8 e = o 8 e ? C . e x p o r t s . u s e L a y o u t E f f e c t : C . e x p o r t s . u s e E f f e c t ; f u n c t i o n s 8 e ( { s t o r e : e , c o n t e x t : t , c h i l d r e n : n , s e r v e r S t a t e : r } ) { c o n s t i = C . e x p o r t s . u s e M e m o ( ( ) = > { c o n s t s = i 8 e ( e ) ; r e t u r n { s t o r e : e , s u b s c r i p t i o n : s , g e t S e r v e r S t a t e : r ? ( ) = > r : v o i d 0 } } , [ e , r ] ) , o = C . e x p o r t s . u s e M e m o ( ( ) = > e . g e t S t a t e ( ) , [ e ] ) ; r e t u r n a 8 e ( ( ) = > { c o n s t { s u b s c r i p t i o n : s } = i ; r e t u r n s . o n S t a t e C h a n g e = s . n o t i f y N e s t e d S u b s , s . t r y S u b s c r i b e ( ) , o ! = = e . g e t S t a t e ( ) & & s . n o t i f y N e s t e d S u b s ( ) , ( ) = > { s . t r y U n s u b s c r i b e ( ) , s . o n S t a t e C h a n g e = v o i d 0 } } , [ i , o ] ) , y ( ( t | | n f ) . P r o v i d e r , { v a l u e : i , c h i l d r e n : n } ) } f u n c t i o n A G ( e = n f ) { c o n s t t = e = = = n f ? P G : ( ) = > C . e x p o r t s . u s e C o n t e x t ( e ) ; r e t u r n f u n c t i o n ( ) { c o n s t { s t o r e : r } = t ( ) ; r e t u r n r } } c o n s t l 8 e = A G ( ) ; f u n c t i o n u 8 e ( e = n f ) { c o n s t t = e = = = n f ? l 8 e : A G ( e ) ; r e t u r n f u n c t i o n ( ) { r e t u r n t ( ) . d i s p a t c h } } c o n s t c 8 e = u 8 e ( ) ; X 9 e ( _ G . e x p o r t s . u s e S y n c E x t e r n a l S t o r e W i t h S e l e c t o r ) ; q 9 e ( p u . e x p o r t s . u n s t a b l e _ b a t c h e d U p d a t e s ) ; f u n c t i o n O 4 ( e ) { r e t u r n t y p e o f S y m b o l = = " f u n c t i o n " & & t y p e o f S y m b o l . i t e r a t o r = = " s y m b o l " ? O 4 = f u n c t i o n ( n ) { r e t u r n t y p e o f n } : O 4 = f u n c t i o n ( n ) { r e t u r n n & & t y p e o f S y m b o l = = " f u n c t i o n " & & n . c o n s t r u c t o r = = = S y m b o l & & n ! = = S y m b o l . p r o t o t y p e ? " s y m b o l " : t y p e o f n } , O 4 ( e ) } f u n c t i o n d 8 e ( e , t ) { i f ( ! ( e i n s t a n c e o f t ) ) t h r o w n e w T y p e E r r o r ( " C a n n o t c a l l a c l a s s a s a f u n c t i o n " ) } f u n c t i o n X I ( e , t ) { f o r ( v a r n = 0 ; n < t
2022-12-26 15:23:16 +00:00
position : relative ;
width : - moz - fit - content ;
width : fit - content ;
height : - moz - fit - content ;
height : fit - content ;
overflow : hidden ;
- webkit - touch - callout : none ; /* iOS Safari */
- webkit - user - select : none ; /* Safari */
- khtml - user - select : none ; /* Konqueror HTML */
- moz - user - select : none ; /* Firefox */
- ms - user - select : none ; /* Internet Explorer/Edge */
user - select : none ;
margin : 0 ;
padding : 0 ;
}
. transform - component - module _content _ _2jYgh {
display : flex ;
flex - wrap : wrap ;
width : - moz - fit - content ;
width : fit - content ;
height : - moz - fit - content ;
height : fit - content ;
margin : 0 ;
padding : 0 ;
transform - origin : 0 % 0 % ;
}
. transform - component - module _content _ _2jYgh img {
pointer - events : none ;
}
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` ,vN={wrapper:"transform-component-module_wrapper__1_Fgj",content:"transform-component-module_content__2jYgh"};aNe(sNe);var lNe=function(e){var t=e.children,n=e.wrapperClass,r=n===void 0?"":n,i=e.contentClass,o=i===void 0?"":i,a=e.wrapperStyle,s=e.contentStyle,l=C.exports.useContext(UY).setComponents,u=C.exports.useRef(null),f=C.exports.useRef(null);return C.exports.useEffect(function(){var p=u.current,m=f.current;p!==null&&m!==null&&l&&l(p,m)},[]),y("div",{ref:u,className:"react-transform-wrapper "+vN.wrapper+" "+r,style:a,children:y("div",{ref:f,className:"react-transform-component "+vN.content+" "+o,style:s,children:t})})};function uNe({image:e,alt:t,ref:n,styleClass:r}){const[i,o]=C.exports.useState(0),[a,s]=C.exports.useState(!1),l=()=>{o(i===-3?0:i-1)},u=()=>{o(i===3?0:i+1)},f=()=>{s(!a)};return y(oNe,{centerOnInit:!0,minScale:.1,initialPositionX:50,initialPositionY:50,children:({zoomIn:p,zoomOut:m,resetTransform:v,centerView:b})=>q(fn,{children:[q("div",{className:"lightbox-image-options",children:[y(Qe,{icon:y(aLe,{}),"aria-label":"Zoom In",tooltip:"Zoom In",onClick:()=>p(),fontSize:20}),y(Qe,{icon:y(sLe,{}),"aria-label":"Zoom Out",tooltip:"Zoom Out",onClick:()=>m(),fontSize:20}),y(Qe,{icon:y(iLe,{}),"aria-label":"Rotate Left",tooltip:"Rotate Left",onClick:l,fontSize:20}),y(Qe,{icon:y(oLe,{}),"aria-label":"Rotate Right",tooltip:"Rotate Right",onClick:u,fontSize:20}),y(Qe,{icon:y(eLe,{}),"aria-label":"Flip Image",tooltip:"Flip Image",onClick:f,fontSize:20}),y(Qe,{icon:y(Ux,{}),"aria-label":"Reset",tooltip:"Reset",onClick:()=>{v(),o(0),s(!1)},fontSize:20})]}),y(lNe,{wrapperStyle:{width:"100%",height:"100%"},children:y("img",{style:{transform: ` rotate ( $ { i * 90 } deg ) scaleX ( $ { a ? - 1 : 1 } ) ` ,width:"100%"},src:e,alt:t,ref:n,className:r||"",onLoad:()=>b(1,0,"easeOut")})})]})})}function cNe(){const e=Be(),t=ge(m=>m.options.isLightBoxOpen),{viewerImageToDisplay:n,shouldShowImageDetails:r,isOnFirstImage:i,isOnLastImage:o}=ge(Jq),[a,s]=C.exports.useState(!1),l=()=>{s(!0)},u=()=>{s(!1)},f=()=>{e(GE())},p=()=>{e(WE())};return Ze("Esc",()=>{t&&e(Qu(!1))},[t]),q("div",{className:"lightbox-container",children:[y(Qe,{icon:y(rLe,{}),"aria-label":"Exit Viewer",className:"lightbox-close-btn",onClick:()=>{e(Qu(!1))},fontSize:20}),q("div",{className:"lightbox-display-container",children:[q("div",{className:"lightbox-preview-wrapper",children:[y(Zq,{}),!r&&q("div",{className:"current-image-next-prev-buttons",children:[y("div",{className:"next-prev-button-trigger-area prev-button-trigger-area",onMouseOver:l,onMouseOut:u,children:a&&!i&&y(hs,{"aria-label":"Previous image",icon:y(DG,{className:"next-prev-button"}),variant:"unstyled",onClick:f})}),y("div",{className:"next-prev-button-trigger-area next-button-trigger-area",onMouseOver:l,onMouseOut:u,children:a&&!o&&y(hs,{"aria-label":"Next image",icon:y(NG,{className:"next-prev-button"}),variant:"unstyled",onClick:p})})]}),n&&q(fn,{children:[y(uNe,{image:n.url,styleClass:"lightbox-image"}),r&&y(Qq,{image:n})]})]}),y(OY,{})]})]})}const dNe=st([hP],e=>{const{seamSize:t,seamBlur:n,seamStrength:r,seamSteps:i}=e;return{seamSize:t,seamBlur:n,seamStrength:r,seamSteps:i}},{memoizeOptions:{resultEqualityCheck:Fe.isEqual}}),fNe=()=>{const e=Be(),{seamSize:t,seamBlur:n,seamStrength:r,seamSteps:i}=ge(dNe),{t:o}=Ye();return q(tt,{direction:"column",gap:"1rem",children:[y(Qo,{sliderMarkRightOffset:-6,label:o("options:seamSize"),min:1,max:256,sliderNumberInputProps:{max:512},value:t,onChange:a=>{e(bR(a))},handleReset:()=>e(bR(96)),withInput:!0,withSliderMarks:!0,withReset:!0}),y(Qo,{sliderMarkRightOffset:-4,label:o("options:seamBlur"),min:0,max:64,sliderNumberInputProps:{max:512},value:n,onChange:a=>{e(yR(a))},handleReset:()=>{e(yR(16))},withInput:!0,withSliderMarks:!0,withReset:!0}),y(Qo,{sliderMarkRightOffset:-7,label:o("options:seamStrength"),min:.01,max:.99,step:.01,value:r,onChange:a=>{e(xR(a))},handleReset:()=>{e(xR(.7))},withInput:!0,withSliderMarks:!0,withReset:!0}),y(Qo,{sliderMarkRightOffset:-4,label:o("options:seamSteps"),min:1,max:32,sliderNumberInputProps:{max:100},value:i,onChange:a=>{e(SR(a))},hand
2022-12-26 15:23:16 +00:00
* @ license React
* react - reconciler . production . min . js
*
* Copyright ( c ) Facebook , Inc . and its affiliates .
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree .
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
* /var xNe=function(t){var n={},r=C.exports,i=fg.exports,o=Object.assign;function a(d){for(var h="https:/ / reactjs . org / docs / error - decoder . html ? invariant = "+d,S=1;S<arguments.length;S++)h+=" & args [ ] = "+encodeURIComponent(arguments[S]);return" Minified React error # "+d+" ; visit "+h+" for the full message or use the non - minified dev environment for full errors and additional helpful warnings . "}var s=r.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED,l=Symbol.for(" react . element "),u=Symbol.for(" react . portal "),f=Symbol.for(" react . fragment "),p=Symbol.for(" react . strict _mode "),m=Symbol.for(" react . profiler "),v=Symbol.for(" react . provider "),b=Symbol.for(" react . context "),x=Symbol.for(" react . forward _ref "),E=Symbol.for(" react . suspense "),k=Symbol.for(" react . suspense _list "),_=Symbol.for(" react . memo "),T=Symbol.for(" react . lazy "),A=Symbol.for(" react . offscreen "),R=Symbol.iterator;function I(d){return d===null||typeof d!=" object "?null:(d=R&&d[R]||d[" @ @ iterator "],typeof d==" function "?d:null)}function D(d){if(d==null)return null;if(typeof d==" function ")return d.displayName||d.name||null;if(typeof d==" string ")return d;switch(d){case f:return" Fragment ";case u:return" Portal ";case m:return" Profiler ";case p:return" StrictMode ";case E:return" Suspense ";case k:return" SuspenseList "}if(typeof d==" object ")switch(d.$$typeof){case b:return(d.displayName||" Context ")+" . Consumer ";case v:return(d._context.displayName||" Context ")+" . Provider ";case x:var h=d.render;return d=d.displayName,d||(d=h.displayName||h.name||" ",d=d!==" "?" ForwardRef ( "+d+" ) ":" ForwardRef "),d;case _:return h=d.displayName||null,h!==null?h:D(d.type)||" Memo ";case T:h=d._payload,d=d._init;try{return D(d(h))}catch(S){}}return null}function B(d){var h=d.type;switch(d.tag){case 24:return" Cache ";case 9:return(h.displayName||" Context ")+" . Consumer ";case 10:return(h._context.displayName||" Context ")+" . Provider ";case 18:return" DehydratedFragment ";case 11:return d=h.render,d=d.displayName||d.name||" ",h.displayName||(d!==" "?" ForwardRef ( "+d+" ) ":" ForwardRef ");case 7:return" Fragment ";case 5:return h;case 4:return" Portal ";case 3:return" Root ";case 6:return" Text ";case 16:return D(h);case 8:return h===p?" StrictMode ":" Mode ";case 22:return" Offscreen ";case 12:return" Profiler ";case 21:return" Scope ";case 13:return" Suspense ";case 19:return" SuspenseList ";case 25:return" TracingMarker ";case 1:case 0:case 17:case 2:case 14:case 15:if(typeof h==" function ")return h.displayName||h.name||null;if(typeof h==" string " ) return h } return null } function z ( d ) { var h = d , S = d ; if ( d . alternate ) for ( ; h . return ; ) h = h . return ; else { d = h ; do h = d , ( h . flags & 4098 ) !== 0 && ( S = h . return ) , d = h . return ; while ( d ) } return h . tag === 3 ? S : null } function j ( d ) { if ( z ( d ) !== d ) throw Error ( a ( 188 ) ) } function K ( d ) { var h = d . alternate ; if ( ! h ) { if ( h = z ( d ) , h === null ) throw Error ( a ( 188 ) ) ; return h !== d ? null : d } for ( var S = d , P = h ; ; ) { var L = S . return ; if ( L === null ) break ; var M = L . alternate ; if ( M === null ) { if ( P = L . return , P !== null ) { S = P ; continue } break } if ( L . child === M . child ) { for ( M = L . child ; M ; ) { if ( M === S ) return j ( L ) , d ; if ( M === P ) return j ( L ) , h ; M = M . sibling } throw Error ( a ( 188 ) ) } if ( S . return !== P . return ) S = L , P = M ; else { for ( var W = ! 1 , se = L . child ; se ; ) { if ( se === S ) { W = ! 0 , S = L , P = M ; break } if ( se === P ) { W = ! 0 , P = L , S = M ; break } se = se . sibling } if ( ! W ) { for ( se = M . child ; se ; ) { if ( se === S ) { W = ! 0 , S = M , P = L ; break } if ( se === P ) { W = ! 0 , P = M , S = L ; break } se = se . sibling } if ( ! W ) throw Error ( a ( 189 ) ) } } if ( S . alternate !== P ) throw Error ( a ( 190 ) ) } if ( S . tag !== 3 ) throw Error ( a ( 188 ) ) ; return S . stateNode . current === S ? d : h } function te ( d ) { return d = K ( d ) , d !== null ? G ( d ) : null } function G ( d ) { if ( d . tag === 5 || d . tag === 6 ) return d ; for ( d = d . child ; d !== null ; ) { var h = G ( d ) ; if ( h !== null ) return h ; d = d . sibling } return null } function $ ( d ) { if ( d . tag === 5 || d . tag === 6 ) return d ; for ( d = d . child ; d !== null ; ) { if ( d . tag !== 4 ) { var h = $ ( d ) ; if ( h !== null ) return h } d = d . sibling } return null } var V = Array . isArray , J = t . getPublicInstance , Z = t . getRootHostContext , U = t . getChildHostContext , X = t . prepareForCommit , le = t . resetAfterCommit , fe = t . createInstance , Se = t . appendInitialChild , Ae = t . finalizeInitialChildren , be = t . prepareUpdate , ke = t . shouldSetTextContent , De = t . createTextInstance , Je = t . scheduleTimeout , Ke = t . cancelTimeout , mt = t . noTimeout , _t = t . isPrimaryRenderer , dt = t . supportsMutation , vt = t . supports
` +xu+d}var ul=!1;function Cu(d,h){if(!d||ul)return"";ul=!0;var S=Error.prepareStackTrace;Error.prepareStackTrace=void 0;try{if(h)if(h=function(){throw Error()},Object.defineProperty(h.prototype,"props",{set:function(){throw Error()}}),typeof Reflect=="object"&&Reflect.construct){try{Reflect.construct(h,[])}catch(He){var P=He}Reflect.construct(d,[],h)}else{try{h.call()}catch(He){P=He}d.call(h.prototype)}else{try{throw Error()}catch(He){P=He}d()}}catch(He){if(He&&P&&typeof He.stack=="string"){for(var L=He.stack.split( `
2022-12-26 15:23:16 +00:00
` ),M=P.stack.split( `
` ),W=L.length-1,se=M.length-1;1<=W&&0<=se&&L[W]!==M[se];)se--;for(;1<=W&&0<=se;W--,se--)if(L[W]!==M[se]){if(W!==1||se!==1)do if(W--,se--,0>se||L[W]!==M[se]){var pe= `
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` +L[W].replace(" at new "," at ");return d.displayName&&pe.includes("<anonymous>")&&(pe=pe.replace("<anonymous>",d.displayName)),pe}while(1<=W&&0<=se);break}}}finally{ul=!1,Error.prepareStackTrace=S}return(d=d?d.displayName||d.name:"")?wu(d):""}var Tp=Object.prototype.hasOwnProperty,kc=[],cl=-1;function sa(d){return{current:d}}function Mn(d){0>cl||(d.current=kc[cl],kc[cl]=null,cl--)}function kn(d,h){cl++,kc[cl]=d.current,d.current=h}var la={},zr=sa(la),si=sa(!1),ua=la;function dl(d,h){var S=d.type.contextTypes;if(!S)return la;var P=d.stateNode;if(P&&P.__reactInternalMemoizedUnmaskedChildContext===h)return P.__reactInternalMemoizedMaskedChildContext;var L={},M;for(M in S)L[M]=h[M];return P&&(d=d.stateNode,d.__reactInternalMemoizedUnmaskedChildContext=h,d.__reactInternalMemoizedMaskedChildContext=L),L}function li(d){return d=d.childContextTypes,d!=null}function ws(){Mn(si),Mn(zr)}function _f(d,h,S){if(zr.current!==la)throw Error(a(168));kn(zr,h),kn(si,S)}function _u(d,h,S){var P=d.stateNode;if(h=h.childContextTypes,typeof P.getChildContext!="function")return S;P=P.getChildContext();for(var L in P)if(!(L in h))throw Error(a(108,B(d)||"Unknown",L));return o({},S,P)}function Cs(d){return d=(d=d.stateNode)&&d.__reactInternalMemoizedMergedChildContext||la,ua=zr.current,kn(zr,d),kn(si,si.current),!0}function kf(d,h,S){var P=d.stateNode;if(!P)throw Error(a(169));S?(d=_u(d,h,ua),P.__reactInternalMemoizedMergedChildContext=d,Mn(si),Mn(zr),kn(zr,d)):Mn(si),kn(si,S)}var Oi=Math.clz32?Math.clz32:Ef,Lp=Math.log,Ap=Math.LN2;function Ef(d){return d>>>=0,d===0?32:31-(Lp(d)/Ap|0)|0}var fl=64,Io=4194304;function hl(d){switch(d&-d){case 1:return 1;case 2:return 2;case 4:return 4;case 8:return 8;case 16:return 16;case 32:return 32;case 64:case 128:case 256:case 512:case 1024:case 2048:case 4096:case 8192:case 16384:case 32768:case 65536:case 131072:case 262144:case 524288:case 1048576:case 2097152:return d&4194240;case 4194304:case 8388608:case 16777216:case 33554432:case 67108864:return d&130023424;case 134217728:return 134217728;case 268435456:return 268435456;case 536870912:return 536870912;case 1073741824:return 1073741824;default:return d}}function ku(d,h){var S=d.pendingLanes;if(S===0)return 0;var P=0,L=d.suspendedLanes,M=d.pingedLanes,W=S&268435455;if(W!==0){var se=W&~L;se!==0?P=hl(se):(M&=W,M!==0&&(P=hl(M)))}else W=S&~L,W!==0?P=hl(W):M!==0&&(P=hl(M));if(P===0)return 0;if(h!==0&&h!==P&&(h&L)===0&&(L=P&-P,M=h&-h,L>=M||L===16&&(M&4194240)!==0))return h;if((P&4)!==0&&(P|=S&16),h=d.entangledLanes,h!==0)for(d=d.entanglements,h&=P;0<h;)S=31-Oi(h),L=1<<S,P|=d[S],h&=~L;return P}function G1(d,h){switch(d){case 1:case 2:case 4:return h+250;case 8:case 16:case 32:case 64:case 128:case 256:case 512:case 1024:case 2048:case 4096:case 8192:case 16384:case 32768:case 65536:case 131072:case 262144:case 524288:case 1048576:case 2097152:return h+5e3;case 4194304:case 8388608:case 16777216:case 33554432:case 67108864:return-1;case 134217728:case 268435456:case 536870912:case 1073741824:return-1;default:return-1}}function Ec(d,h){for(var S=d.suspendedLanes,P=d.pingedLanes,L=d.expirationTimes,M=d.pendingLanes;0<M;){var W=31-Oi(M),se=1<<W,pe=L[W];pe===-1?((se&S)===0||(se&P)!==0)&&(L[W]=G1(se,h)):pe<=h&&(d.expiredLanes|=se),M&=~se}}function Pc(d){return d=d.pendingLanes&-1073741825,d!==0?d:d&1073741824?1073741824:0}function Pf(){var d=fl;return fl<<=1,(fl&4194240)===0&&(fl=64),d}function Tc(d){for(var h=[],S=0;31>S;S++)h.push(d);return h}function Ha(d,h,S){d.pendingLanes|=h,h!==536870912&&(d.suspendedLanes=0,d.pingedLanes=0),d=d.eventTimes,h=31-Oi(h),d[h]=S}function Tf(d,h){var S=d.pendingLanes&~h;d.pendingLanes=h,d.suspendedLanes=0,d.pingedLanes=0,d.expiredLanes&=h,d.mutableReadLanes&=h,d.entangledLanes&=h,h=d.entanglements;var P=d.eventTimes;for(d=d.expirationTimes;0<S;){var L=31-Oi(S),M=1<<L;h[L]=0,P[L]=-1,d[L]=-1,S&=~M}}function Eu(d,h){var S=d.entangledLanes|=h;for(d=d.entanglements;S;){var P=31-Oi(S),L=1<<P;L&h|d[P]&h&&(d[P]|=h),S&=~L}}var Xt=0;function Op(d){return d&=-d,1<d?4<d?(d&268435455)!==0?16:536870912:4:1}var Lc=i.unstable_scheduleCallback,Ac
2022-12-26 15:23:16 +00:00
Error generating stack : ` +M.message+ `
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` +M.stack}return{value:d,source:h,stack:L,digest:null}}function um(d,h,S){return{value:d,source:null,stack:S!=null?S:null,digest:h!=null?h:null}}function Kc(d,h){try{console.error(h.value)}catch(S){setTimeout(function(){throw S})}}var qf=typeof WeakMap=="function"?WeakMap:Map;function cm(d,h,S){S=ks(-1,S),S.tag=3,S.payload={element:null};var P=h.value;return S.callback=function(){Zf||(Zf=!0,Qf=P),Kc(d,h)},S}function F(d,h,S){S=ks(-1,S),S.tag=3;var P=d.type.getDerivedStateFromError;if(typeof P=="function"){var L=h.value;S.payload=function(){return P(L)},S.callback=function(){Kc(d,h)}}var M=d.stateNode;return M!==null&&typeof M.componentDidCatch=="function"&&(S.callback=function(){Kc(d,h),typeof P!="function"&&(_l===null?_l=new Set([this]):_l.add(this));var W=h.stack;this.componentDidCatch(h.value,{componentStack:W!==null?W:""})}),S}function Ou(d,h,S){var P=d.pingCache;if(P===null){P=d.pingCache=new qf;var L=new Set;P.set(h,L)}else L=P.get(h),L===void 0&&(L=new Set,P.set(h,L));L.has(S)||(L.add(S),d=m3.bind(null,d,h,S),h.then(d,d))}function Yf(d){do{var h;if((h=d.tag===13)&&(h=d.memoizedState,h=h!==null?h.dehydrated!==null:!0),h)return d;d=d.return}while(d!==null);return null}function po(d,h,S,P,L){return(d.mode&1)===0?(d===h?d.flags|=65536:(d.flags|=128,S.flags|=131072,S.flags&=-52805,S.tag===1&&(S.alternate===null?S.tag=17:(h=ks(-1,1),h.tag=2,ml(S,h,1))),S.lanes|=1),d):(d.flags|=65536,d.lanes=L,d)}var Jt=s.ReactCurrentOwner,Ji=!1;function Di(d,h,S,P){h.child=d===null?Xy(h,null,S,P):Nc(h,d.child,S,P)}function i3(d,h,S,P,L){S=S.render;var M=h.ref;return Ic(h,L),P=nm(d,h,S,P,M,L),S=Fc(),d!==null&&!Ji?(h.updateQueue=d.updateQueue,h.flags&=-2053,d.lanes&=~L,Ps(d,h,L)):(Hn&&S&&Mi(h),h.flags|=1,Di(d,h,P,L),h.child)}function yl(d,h,S,P,L){if(d===null){var M=S.type;return typeof M=="function"&&!Om(M)&&M.defaultProps===void 0&&S.compare===null&&S.defaultProps===void 0?(h.tag=15,h.type=M,o3(d,h,M,P,L)):(d=nh(S.type,null,P,h,h.mode,L),d.ref=h.ref,d.return=h,h.child=d)}if(M=d.child,(d.lanes&L)===0){var W=M.memoizedProps;if(S=S.compare,S=S!==null?S:Pu,S(W,P)&&d.ref===h.ref)return Ps(d,h,L)}return h.flags|=1,d=vo(M,P),d.ref=h.ref,d.return=h,h.child=d}function o3(d,h,S,P,L){if(d!==null){var M=d.memoizedProps;if(Pu(M,P)&&d.ref===h.ref)if(Ji=!1,h.pendingProps=P=M,(d.lanes&L)!==0)(d.flags&131072)!==0&&(Ji=!0);else return h.lanes=d.lanes,Ps(d,h,L)}return dm(d,h,S,P,L)}function a3(d,h,S){var P=h.pendingProps,L=P.children,M=d!==null?d.memoizedState:null;if(P.mode==="hidden")if((h.mode&1)===0)h.memoizedState={baseLanes:0,cachePool:null,transitions:null},kn(xl,go),go|=S;else{if((S&1073741824)===0)return d=M!==null?M.baseLanes|S:S,h.lanes=h.childLanes=1073741824,h.memoizedState={baseLanes:d,cachePool:null,transitions:null},h.updateQueue=null,kn(xl,go),go|=d,null;h.memoizedState={baseLanes:0,cachePool:null,transitions:null},P=M!==null?M.baseLanes:S,kn(xl,go),go|=P}else M!==null?(P=M.baseLanes|S,h.memoizedState=null):P=S,kn(xl,go),go|=P;return Di(d,h,L,S),h.child}function s3(d,h){var S=h.ref;(d===null&&S!==null||d!==null&&d.ref!==S)&&(h.flags|=512,h.flags|=2097152)}function dm(d,h,S,P,L){var M=li(S)?ua:zr.current;return M=dl(h,M),Ic(h,L),S=nm(d,h,S,P,M,L),P=Fc(),d!==null&&!Ji?(h.updateQueue=d.updateQueue,h.flags&=-2053,d.lanes&=~L,Ps(d,h,L)):(Hn&&P&&Mi(h),h.flags|=1,Di(d,h,S,L),h.child)}function l3(d,h,S,P,L){if(li(S)){var M=!0;Cs(h)}else M=!1;if(Ic(h,L),h.stateNode===null)qa(d,h),qy(h,S,P),Q1(h,S,P,L),P=!0;else if(d===null){var W=h.stateNode,se=h.memoizedProps;W.props=se;var pe=W.context,He=S.contextType;typeof He=="object"&&He!==null?He=ho(He):(He=li(S)?ua:zr.current,He=dl(h,He));var ht=S.getDerivedStateFromProps, $ t=typeof ht=="function"||typeof W.getSnapshotBeforeUpdate=="function"; $ t||typeof W.UNSAFE_componentWillReceiveProps!="function"&&typeof W.componentWillReceiveProps!="function"||(se!==P||pe!==He)&&Yy(h,W,P,He),_s=!1;var it=h.memoizedState;W.state=it,Fp(h,P,W,L),pe=h.memoizedState,se!==P||it!==pe||si.current||_s?(typeof ht=="function"&&(jp(h,S,ht,P),pe=h.memoizedState),(se=_s|| $ f(h,S,se,P,it,pe,He))?( $ t||typeof W.UNSAFE_componentWillMoun
2022-12-26 15:23:16 +00:00
` +(P.join(" > ")+ `
No matching component was found for :
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` )+d.join(" > ")}return null},n.getPublicRootInstance=function(d){if(d=d.current,!d.child)return null;switch(d.child.tag){case 5:return J(d.child.stateNode);default:return d.child.stateNode}},n.injectIntoDevTools=function(d){if(d={bundleType:d.bundleType,version:d.version,rendererPackageName:d.rendererPackageName,rendererConfig:d.rendererConfig,overrideHookState:null,overrideHookStateDeletePath:null,overrideHookStateRenamePath:null,overrideProps:null,overridePropsDeletePath:null,overridePropsRenamePath:null,setErrorHandler:null,setSuspenseHandler:null,scheduleUpdate:null,currentDispatcherRef:s.ReactCurrentDispatcher,findHostInstanceByFiber:y0,findFiberByHostInstance:d.findFiberByHostInstance||Im,findHostInstancesForRefresh:null,scheduleRefresh:null,scheduleRoot:null,setRefreshHandler:null,getCurrentFiber:null,reconcilerVersion:"18.2.0"},typeof __REACT_DEVTOOLS_GLOBAL_HOOK__>"u")d=!1;else{var h=__REACT_DEVTOOLS_GLOBAL_HOOK__;if(h.isDisabled||!h.supportsFiber)d=!0;else{try{mn=h.inject(d),Zt=h}catch(S){}d=!!h.checkDCE}}return d},n.isAlreadyRendering=function(){return!1},n.observeVisibleRects=function(d,h,S,P){if(!ue)throw Error(a(363));d=wm(d,h);var L=Nt(d,S,P).disconnect;return{disconnect:function(){L()}}},n.registerMutableSourceForHydration=function(d,h){var S=h._getVersion;S=S(h._source),d.mutableSourceEagerHydrationData==null?d.mutableSourceEagerHydrationData=[h,S]:d.mutableSourceEagerHydrationData.push(h,S)},n.runWithPriority=function(d,h){var S=Xt;try{return Xt=d,h()}finally{Xt=S}},n.shouldError=function(){return null},n.shouldSuspend=function(){return!1},n.updateContainer=function(d,h,S,P){var L=h.current,M=_i(),W=Wr(L);return S=Mm(S),h.context===null?h.context=S:h.pendingContext=S,h=ks(M,W),h.payload={element:d},P=P===void 0?null:P,P!==null&&(h.callback=P),d=ml(L,h,W),d!==null&&(jo(d,L,W,M), $ p(d,L,W)),W},n};(function(e){e.exports=xNe})(GY);const wNe=t_(GY.exports);var ZP={exports:{}},_p={};/**
2022-12-26 15:23:16 +00:00
* @ license React
* react - reconciler - constants . production . min . js
*
* Copyright ( c ) Facebook , Inc . and its affiliates .
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree .
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
* / _ p . C o n c u r r e n t R o o t = 1 ; _ p . C o n t i n u o u s E v e n t P r i o r i t y = 4 ; _ p . D e f a u l t E v e n t P r i o r i t y = 1 6 ; _ p . D i s c r e t e E v e n t P r i o r i t y = 1 ; _ p . I d l e E v e n t P r i o r i t y = 5 3 6 8 7 0 9 1 2 ; _ p . L e g a c y R o o t = 0 ; ( f u n c t i o n ( e ) { e . e x p o r t s = _ p } ) ( Z P ) ; c o n s t y N = { c h i l d r e n : ! 0 , r e f : ! 0 , k e y : ! 0 , s t y l e : ! 0 , f o r w a r d e d R e f : ! 0 , u n s t a b l e _ a p p l y C a c h e : ! 0 , u n s t a b l e _ a p p l y D r a w H i t F r o m C a c h e : ! 0 } ; l e t b N = ! 1 , S N = ! 1 ; c o n s t Q P = " . r e a c t - k o n v a - e v e n t " , C N e = ` R e a c t K o n v a : Y o u h a v e a K o n v a n o d e w i t h d r a g g a b l e = t r u e a n d p o s i t i o n d e f i n e d b u t n o o n D r a g M o v e o r o n D r a g E n d e v e n t s a r e h a n d l e d .
2022-12-26 15:23:16 +00:00
Position of a node will be changed during drag & drop , so you should update state of the react app as well .
Consider to add onDragMove or onDragEnd events .
For more info see : https : //github.com/konvajs/react-konva/issues/256
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` ,_Ne= ` ReactKonva : You are using "zIndex" attribute for a Konva node .
2022-12-26 15:23:16 +00:00
react - konva may get confused with ordering . Just define correct order of elements in your render function of a component .
For more info see : https : //github.com/konvajs/react-konva/issues/194
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
` ,kNe={};function aw(e,t,n=kNe){if(!bN&&"zIndex"in t&&(console.warn(_Ne),bN=!0),!SN&&t.draggable){var r=t.x!==void 0||t.y!==void 0,i=t.onDragEnd||t.onDragMove;r&&!i&&(console.warn(CNe),SN=!0)}for(var o in n)if(!yN[o]){var a=o.slice(0,2)==="on",s=n[o]!==t[o];if(a&&s){var l=o.substr(2).toLowerCase();l.substr(0,7)==="content"&&(l="content"+l.substr(7,1).toUpperCase()+l.substr(8)),e.off(l,n[o])}var u=!t.hasOwnProperty(o);u&&e.setAttr(o,void 0)}var f=t._useStrictMode,p={},m=!1;const v={};for(var o in t)if(!yN[o]){var a=o.slice(0,2)==="on",b=n[o]!==t[o];if(a&&b){var l=o.substr(2).toLowerCase();l.substr(0,7)==="content"&&(l="content"+l.substr(7,1).toUpperCase()+l.substr(8)),t[o]&&(v[l]=t[o])}!a&&(t[o]!==n[o]||f&&t[o]!==e.getAttr(o))&&(m=!0,p[o]=t[o])}m&&(e.setAttrs(p),xf(e));for(var l in v)e.on(l+QP,v[l])}function xf(e){if(!gt.autoDrawEnabled){var t=e.getLayer()||e.getStage();t&&t.batchDraw()}}const qY={},ENe={};rp.Node.prototype._applyProps=aw;function PNe(e,t){if(typeof t=="string"){console.error( ` Do not use plain text as child of Konva . Node . You are using text : $ { t } ` );return}e.add(t),xf(e)}function TNe(e,t,n){let r=rp[e];r||(console.error( ` Konva has no node with the type $ { e } . Group will be used instead . If you use minimal version of react - konva , just import required nodes into Konva : "import " konva / lib / shapes / $ { e } " If you want to render DOM elements as part of canvas tree take a look into this demo: https://konvajs.github.io/docs/react/DOM_Portal.html`),r=rp.Group);const i={},o={};for(var a in t){var s=a.slice(0,2)===" on ";s?o[a]=t[a]:i[a]=t[a]}const l=new r(i);return aw(l,o),l}function LNe(e,t,n){console.error(`Text components are not supported for now in ReactKonva. Your text is: " $ { e } "`)}function ANe(e,t,n){return!1}function ONe(e){return e}function MNe(){return null}function RNe(){return null}function INe(e,t,n,r){return ENe}function DNe(){}function NNe(e){}function BNe(e,t){return!1}function $Ne(){return qY}function FNe(){return qY}const zNe=setTimeout,jNe=clearTimeout,HNe=-1;function VNe(e,t){return!1}const UNe=!1,WNe=!0,GNe=!0;function qNe(e,t){t.parent===e?t.moveToTop():e.add(t),xf(e)}function YNe(e,t){t.parent===e?t.moveToTop():e.add(t),xf(e)}function YY(e,t,n){t._remove(),e.add(t),t.setZIndex(n.getZIndex()),xf(e)}function KNe(e,t,n){YY(e,t,n)}function XNe(e,t){t.destroy(),t.off(QP),xf(e)}function ZNe(e,t){t.destroy(),t.off(QP),xf(e)}function QNe(e,t,n){console.error(`Text components are not yet supported in ReactKonva. You text is: " $ { n } "`)}function JNe(e,t,n){}function eBe(e,t,n,r,i){aw(e,i,r)}function tBe(e){e.hide(),xf(e)}function nBe(e){}function rBe(e,t){(t.visible==null||t.visible)&&e.show()}function iBe(e,t){}function oBe(e){}function aBe(){}const sBe=()=>ZP.exports.DefaultEventPriority,lBe=Object.freeze(Object.defineProperty({__proto__:null,appendInitialChild:PNe,createInstance:TNe,createTextInstance:LNe,finalizeInitialChildren:ANe,getPublicInstance:ONe,prepareForCommit:MNe,preparePortalMount:RNe,prepareUpdate:INe,resetAfterCommit:DNe,resetTextContent:NNe,shouldDeprioritizeSubtree:BNe,getRootHostContext:$Ne,getChildHostContext:FNe,scheduleTimeout:zNe,cancelTimeout:jNe,noTimeout:HNe,shouldSetTextContent:VNe,isPrimaryRenderer:UNe,warnsIfNotActing:WNe,supportsMutation:GNe,appendChild:qNe,appendChildToContainer:YNe,insertBefore:YY,insertInContainerBefore:KNe,removeChild:XNe,removeChildFromContainer:ZNe,commitTextUpdate:QNe,commitMount:JNe,commitUpdate:eBe,hideInstance:tBe,hideTextInstance:nBe,unhideInstance:rBe,unhideTextInstance:iBe,clearContainer:oBe,detachDeletedInstance:aBe,getCurrentEventPriority:sBe,now:fg.exports.unstable_now,idlePriority:fg.exports.unstable_IdlePriority,run:fg.exports.unstable_runWithPriority},Symbol.toStringTag,{value:" Module " } ) ) ; var uBe = Object . defineProperty , cBe = Object . defineProperties , dBe = Object . getOwnPropertyDescriptors , xN = Object . getOwnPropertySymbols , fBe = Object . prototype . hasOwnProperty , hBe = Object . prototype . propertyIsEnumerable , wN = ( e , t , n ) => t in e ? uBe ( e , t , { enumerable : ! 0 , configurable : ! 0 , writable : ! 0 , value : n } ) : e [ t ] = n , CN = ( e , t ) => { for ( var n in t || ( t = { } ) ) fBe . call ( t , n ) && wN ( e , n , t [ n ] ) ; if ( xN )
2022-12-26 15:23:16 +00:00
< ! DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd" >
< svg width = "60px" height = "60px" viewBox = "0 0 30 30" version = "1.1" xmlns = "http://www.w3.org/2000/svg" xmlns : xlink = "http://www.w3.org/1999/xlink" xml : space = "preserve" xmlns : serif = "http://www.serif.com/" style = "fill-rule:evenodd;clip-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5;" >
< g transform = "matrix(0.5,0,0,0.5,0,0)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,2.5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,7.5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,10)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,12.5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,15)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,17.5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,20)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,22.5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,25)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,27.5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,30)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,-2.5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,-5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,-7.5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,-10)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,-12.5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,-15)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,-17.5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,-20)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,-22.5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,-25)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,-27.5)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
< g transform = "matrix(0.5,0,0,0.5,0,-30)" >
< path d = "M-3.5,63.5L64,-4" style = "fill:none;stroke:black;stroke-width:1px;" / >
< / g >
use 🧨diffusers model (#1583)
* initial commit of DiffusionPipeline class
* spike: proof of concept using diffusers for txt2img
* doc: type hints for Generator
* refactor(model_cache): factor out load_ckpt
* model_cache: add ability to load a diffusers model pipeline
and update associated things in Generate & Generator to not instantly fail when that happens
* model_cache: fix model default image dimensions
* txt2img: support switching diffusers schedulers
* diffusers: let the scheduler do its scaling of the initial latents
Remove IPNDM scheduler; it is not behaving.
* web server: update image_progress callback for diffusers data
* diffusers: restore prompt weighting feature
* diffusers: fix set-sampler error following model switch
* diffusers: use InvokeAIDiffuserComponent for conditioning
* cross_attention_control: stub (no-op) implementations for diffusers
* model_cache: let offload_model work with DiffusionPipeline, sorta.
* models.yaml.example: add diffusers-format model, set as default
* test-invoke-conda: use diffusers-format model
test-invoke-conda: put huggingface-token where the library can use it
* environment-mac: upgrade to diffusers 0.7 (from 0.6)
this was already done for linux; mac must have been lost in the merge.
* preload_models: explicitly load diffusers models
In non-interactive mode too, as long as you're logged in.
* fix(model_cache): don't check `model.config` in diffusers format
clean-up from recent merge.
* diffusers integration: support img2img
* dev: upgrade to diffusers 0.8 (from 0.7.1)
We get to remove some code by using methods that were factored out in the base class.
* refactor: remove backported img2img.get_timesteps
now that we can use it directly from diffusers 0.8.1
* ci: use diffusers model
* dev: upgrade to diffusers 0.9 (from 0.8.1)
* lint: correct annotations for Python 3.9.
* lint: correct AttributeError.name reference for Python 3.9.
* CI: prefer diffusers-1.4 because it no longer requires a token
The RunwayML models still do.
* build: there's yet another place to update requirements?
* configure: try to download models even without token
Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.)
* configure: add troubleshooting info for config-not-found
* fix(configure): prepend root to config path
* fix(configure): remove second `default: true` from models example
* CI: simplify test-on-push logic now that we don't need secrets
The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks.
* create an embedding_manager for diffusers
* internal: avoid importing diffusers DummyObject
see https://github.com/huggingface/diffusers/issues/1479
* fix "config attributes…not expected" diffusers warnings.
* fix deprecated scheduler construction
* work around an apparent MPS torch bug that causes conditioning to have no effect
* 🚧 post-rebase repair
* preliminary support for outpainting (no masking yet)
* monkey-patch diffusers.attention and use Invoke lowvram code
* add always_use_cpu arg to bypass MPS
* add cross-attention control support to diffusers (fails on MPS)
For unknown reasons MPS produces garbage output with .swap(). Use
--always_use_cpu arg to invoke.py for now to test this code on MPS.
* diffusers support for the inpainting model
* fix debug_image to not crash with non-RGB images.
* inpainting for the normal model [WIP]
This seems to be performing well until the LAST STEP, at which point it dissolves to confetti.
* fix off-by-one bug in cross-attention-control (#1774)
prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness).
based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly.
* refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary
* inpainting for the normal model. I think it works this time.
* diffusers: reset num_vectors_per_token
sync with 44a00555718f1df173c60da0ed646cf700e29537
* diffusers: txt2img2img (hires_fix)
with so much slicing and dicing of pipeline methods to stitch them together
* refactor(diffusers): reduce some code duplication amongst the different tasks
* fixup! refactor(diffusers): reduce some code duplication amongst the different tasks
* diffusers: enable DPMSolver++ scheduler
* diffusers: upgrade to diffusers 0.10, add Heun scheduler
* diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers
* CI: default to diffusers-1.5 now that runwayml token requirement is gone
* diffusers: update to 0.10 (and transformers to 4.25)
* diffusers: use xformers when available
diffusers no longer auto-enables this as of 0.10.2.
* diffusers: make masked img2img behave better with multi-step schedulers
re-randomizing the noise each step was confusing them.
* diffusers: work more better with more models.
fixed relative path problem with local models.
fixed models on hub not always having a `fp16` branch.
* diffusers: stopgap fix for attention_maps_callback crash after recent merge
* fixup import merge conflicts
correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b
* test: add tests/inpainting inputs for masked img2img
* diffusers(AddsMaskedGuidance): partial fix for k-schedulers
Prevents them from crashing, but results are still hot garbage.
* fix --safety_checker arg parsing
and add note to diffusers loader about where safety checker gets called
* generate: fix import error
* CI: don't try to read the old init location
* diffusers: support loading an alternate VAE
* CI: remove sh-syntax if-statement so it doesn't crash powershell
* CI: fold strings in yaml because backslash is not line-continuation in powershell
* attention maps callback stuff for diffusers
* build: fix syntax error in environment-mac
* diffusers: add INITIAL_MODELS with diffusers-compatible repos
* re-enable the embedding manager; closes #1778
* Squashed commit of the following:
commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:43:07 2022 +0100
import new load handling from EmbeddingManager and cleanup
commit c4abe91a5ba0d415b45bf734068385668b7a66e6
Merge: 032e856e 1efc6397
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:09:53 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager
commit 032e856eefb3bbc39534f5daafd25764bcfcef8b
Merge: 8b4f0fe9 bc515e24
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:08:01 2022 +0100
Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager
commit 1efc6397fc6e61c1aff4b0258b93089d61de5955
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 15:04:28 2022 +0100
cleanup and add performance notes
commit e400f804ac471a0ca2ba432fd658778b20c7bdab
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:45:07 2022 +0100
fix bug and update unit tests
commit deb9ae0ae1016750e93ce8275734061f7285a231
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 14:28:29 2022 +0100
textual inversion manager seems to work
commit 162e02505dec777e91a983c4d0fb52e950d25ff0
Merge: cbad4583 12769b3d
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:58:03 2022 +0100
Merge branch 'main' into feature_textual_inversion_mgr
commit cbad45836c6aace6871a90f2621a953f49433131
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:54:10 2022 +0100
use position embeddings
commit 070344c69b0e0db340a183857d0a787b348681d3
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:53:47 2022 +0100
Don't crash CLI on exceptions
commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8
Author: Damian Stewart <d@damianstewart.com>
Date: Sun Dec 18 11:11:55 2022 +0100
add missing position_embeddings
commit 12769b3d3562ef71e0f54946b532ad077e10043c
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:33:25 2022 +0100
debugging why it don't work
commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 13:21:33 2022 +0100
debugging why it don't work
commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 12:48:38 2022 +0100
use TextualInversionManager in place of embeddings (wip, doesn't work)
commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf
Merge: 6e4dad60 023df37e
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:37:31 2022 +0100
Merge branch 'feature_textual_inversion_mgr' into dev/diffusers
commit 023df37efffa67434f77def7fc3c9dfb29f699fd
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:36:54 2022 +0100
cleanup
commit 05fac594eaf79d0058e3c48deee93df603f136c2
Author: Damian Stewart <d@damianstewart.com>
Date: Fri Dec 16 02:07:49 2022 +0100
tweak error checking
commit 009f32ed39a7280997c3ffab112adadee0b44279
Author: damian <null@damianstewart.com>
Date: Thu Dec 15 21:29:47 2022 +0100
unit tests passing for embeddings with vector length >1
commit beb1b08d9a98112ed2fe073580568e1a18698da3
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:39:09 2022 +0100
more explicit equality tests when overwriting
commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 13:30:13 2022 +0100
wip textual inversion manager (unit tests passing for 1v embedding overwriting)
commit 417c2b57d90924a839616bfb66804faab8039e4c
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 12:30:55 2022 +0100
wip textual inversion manager (unit tests passing for base stuff + padding)
commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff
Author: Damian Stewart <d@damianstewart.com>
Date: Thu Dec 15 10:57:57 2022 +0100
wip new TextualInversionManager
* stop using WeightedFrozenCLIPEmbedder
* store diffusion models locally
- configure_invokeai.py reconfigured to store diffusion models rather than
CompVis models
- hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id
- models.yaml does **NOT** use path, just repo_id
- "repo_name" changed to "repo_id" to following hugging face conventions
- Models are loaded with full precision pending further work.
* allow non-local files during development
* path takes priority over repo_id
* MVP for model_cache and configure_invokeai
- Feature complete (almost)
- configure_invokeai.py downloads both .ckpt and diffuser models,
along with their VAEs. Both types of download are controlled by
a unified INITIAL_MODELS.yaml file.
- model_cache can load both type of model and switches back and forth
in CPU. No memory leaks detected
TO DO:
1. I have not yet turned on the LocalOnly flag for diffuser models, so
the code will check the Hugging Face repo for updates before using the
locally cached models. This will break firewalled systems. I am thinking
of putting in a global check for internet connectivity at startup time
and setting the LocalOnly flag based on this. It would be good to check
updates if there is connectivity.
2. I have not gone completely through INITIAL_MODELS.yaml to check which
models are available as diffusers and which are not. So models like
PaperCut and VoxelArt may not load properly. The runway and stability
models are checked, as well as the Trinart models.
3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml
REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE:
1. When loading a .ckpt file there are lots of messages like this:
Warning! ldm.modules.attention.CrossAttention is no longer being
maintained. Please use InvokeAICrossAttention instead.
I'm not sure how to address this.
2. The ckpt models ***don't actually run*** due to the lack of special-case
support for them in the generator objects. For example, here's the hard
crash you get when you run txt2img against the legacy waifu-diffusion-1.3
model:
```
>> An error occurred:
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main
main_loop(gen, opt)
File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop
gen.prompt2image(
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings'
```
3. The inpainting diffusion model isn't working. Here's the output of "banana
sushi" when inpainting-1.5 is loaded:
```
Traceback (most recent call last):
File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image
results = generator.generate(
File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate
image = make_image(x_T)
File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image
pipeline_output = pipeline.image_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings
result_latents, result_attention_map_saver = self.latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings
result: PipelineIntermediateState = infer_latents_from_embeddings(
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__
for result in self.generator_method(*args, **kwargs):
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings
step_output = self.step(batched_t, latents, guidance_scale,
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step
step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs)
File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step
pred_original_sample = sample - sigma * model_output
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1
```
* proper support for float32/float16
- configure script now correctly detects user's preference for
fp16/32 and downloads the correct diffuser version. If fp16
version not available, falls back to fp32 version.
- misc code cleanup and simplification in model_cache
* add on-the-fly conversion of .ckpt to diffusers models
1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py.
2. A new !optimize command has been added to the CLI. Should be ported to Web GUI.
User experience on the CLI is this:
```
invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt
INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model.
This operation will take 30-60s to complete.
Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
Writing new config file entry for sd-v1-4...
>> New configuration:
sd-v1-4:
description: Optimized version of sd-v1-4
format: diffusers
path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
OK to import [n]? y
>> Verifying that new model loads...
>> Current VRAM usage: 2.60G
>> Offloading stable-diffusion-2.1 to CPU
>> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4
| Using faster float16 precision
You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \
license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\
disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
| training width x height = (512 x 512)
>> Model loaded in 3.48s
>> Max VRAM used to load the model: 2.17G
>> Current VRAM usage:2.17G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
Keep model loaded? [y]
```
* add parallel set of generator files for ckpt legacy generation
* generation using legacy ckpt models now working
* diffusers: fix missing attention_maps_callback
fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36
* associate legacy CrossAttention with .ckpt models
* enable autoconvert
New --autoconvert CLI option will scan a designated directory for
new .ckpt files, convert them into diffuser models, and import
them into models.yaml.
Works like this:
invoke.py --autoconvert /path/to/weights/directory
In ModelCache added two new methods:
autoconvert_weights(config_path, weights_directory_path, models_directory_path)
convert_and_import(ckpt_path, diffuser_path)
* diffusers: update to diffusers 0.11 (from 0.10.2)
* fix vae loading & width/height calculation
* refactor: encapsulate these conditioning data into one container
* diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function
* add support for safetensors and accelerate
* set local_files_only when internet unreachable
* diffusers: fix error-handling path when model repo has no fp16 branch
* fix generatorinpaint error
Fixes :
"ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint'
https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318
* quench diffuser safety-checker warning
* diffusers: support stochastic DDIM eta parameter
* fix conda env creation on macos
* fix cross-attention with diffusers 0.11
* diffusers: the VAE needs to be tiling as well as the U-Net
* diffusers: comment on subfolders
* diffusers: embiggen!
* diffusers: make model_cache.list_models serializable
* diffusers(inpaint): restore scaling functionality
* fix requirements clash between numba and numpy 1.24
* diffusers: allow inpainting model to do non-inpainting tasks
* start expanding model_cache functionality
* add import_ckpt_model() and import_diffuser_model() methods to model_manager
- in addition, model_cache.py is now renamed to model_manager.py
* allow "recommended" flag to be optional in INITIAL_MODELS.yaml
* configure_invokeai now downloads VAE diffusers in advance
* rename ModelCache to ModelManager
* remove support for `repo_name` in models.yaml
* check for and refuse to load embeddings trained on incompatible models
* models.yaml.example: s/repo_name/repo_id
and remove extra INITIAL_MODELS now that the main one has diffusers models in it.
* add MVP textual inversion script
* refactor(InvokeAIDiffuserComponent): factor out _combine()
* InvokeAIDiffuserComponent: implement threshold
* InvokeAIDiffuserComponent: diagnostic logs for threshold
...this does not look right
* add a curses-based frontend to textual inversion
- not quite working yet
- requires npyscreen installed
- on windows will also have the windows-curses requirement, but not added
to requirements yet
* add curses-based interface for textual inversion
* fix crash in convert_and_import()
- This corrects a "local variable referenced before assignment" error
in model_manager.convert_and_import()
* potential workaround for no 'state_dict' key error
- As reported in https://github.com/huggingface/diffusers/issues/1876
* create TI output dir if needed
* Update environment-lin-cuda.yml (#2159)
Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~
* diffusers: update sampler-to-scheduler mapping
based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672
* improve user exp for ckt to diffusers conversion
- !optimize_models command now operates on an existing ckpt file entry in models.yaml
- replaces existing entry, rather than adding a new one
- offers to delete the ckpt file after conversion
* web: adapt progress callback to deal with old generator or new diffusers pipeline
* clean-up model_manager code
- add_model() verified to work for .ckpt local paths,
.ckpt remote URLs, diffusers local paths, and
diffusers repo_ids
- convert_and_import() verified to work for local and
remove .ckpt files
* handle edge cases for import_model() and convert_model()
* add support for safetensor .ckpt files
* fix name error
* code cleanup with pyflake
* improve model setting behavior
- If the user enters an invalid model name at startup time, will not
try to load it, warn, and use default model
- CLI UI enhancement: include currently active model in the command
line prompt.
* update test-invoke-pip.yml
- fix model cache path to point to runwayml/stable-diffusion-v1-5
- remove `skip-sd-weights` from configure_invokeai.py args
* exclude dev/diffusers from "fail for draft PRs"
* disable "fail on PR jobs"
* re-add `--skip-sd-weights` since no space
* update workflow environments
- include `INVOKE_MODEL_RECONFIGURE: '--yes'`
* clean up model load failure handling
- Allow CLI to run even when no model is defined or loadable.
- Inhibit stack trace when model load fails - only show last error
- Give user *option* to run configure_invokeai.py when no models
successfully load.
- Restart invokeai after reconfiguration.
* further edge-case handling
1) only one model in models.yaml file, and that model is broken
2) no models in models.yaml
3) models.yaml doesn't exist at all
* fix incorrect model status listing
- "cached" was not being returned from list_models()
- normalize handling of exceptions during model loading:
- Passing an invalid model name to generate.set_model() will return
a KeyError
- All other exceptions are returned as the appropriate Exception
* CI: do download weights (if not already cached)
* diffusers: fix scheduler loading in offline mode
* CI: fix model name (no longer has `diffusers-` prefix)
* Update txt2img2img.py (#2256)
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* feat - make model storage compatible with hugging face caching system
This commit alters the InvokeAI model directory to be compatible with
hugging face, making it easier to share diffusers (and other models)
across different programs.
- If the HF_HOME environment variable is not set, then models are
cached in ~/invokeai/models in a format that is identical to the
HuggingFace cache.
- If HF_HOME is set, then models are cached wherever HF_HOME points.
- To enable sharing with other HuggingFace library clients, set
HF_HOME to ~/.cache/huggingface to set the default cache location
or to ~/invokeai/models to have huggingface cache inside InvokeAI.
* fixes to share models with HuggingFace cache system
- If HF_HOME environment variable is defined, then all huggingface models
are stored in that directory following the standard conventions.
- For seamless interoperability, set HF_HOME to ~/.cache/huggingface
- If HF_HOME not defined, then models are stored in ~/invokeai/models.
This is equivalent to setting HF_HOME to ~/invokeai/models
A future commit will add a migration mechanism so that this change doesn't
break previous installs.
* fix error "no attribute CkptInpaint"
* model_manager.list_models() returns entire model config stanza+status
* Initial Draft - Model Manager Diffusers
* added hash function to diffusers
* implement sha256 hashes on diffusers models
* Add Model Manager Support for Diffusers
* fix various problems with model manager
- in cli import functions, fix not enough values to unpack from
_get_name_and_desc()
- fix crash when using old-style vae: value with new-style diffuser
* rebuild frontend
* fix dictconfig-not-serializable issue
* fix NoneType' object is not subscriptable crash in model_manager
* fix "str has no attribute get" error in model_manager list_models()
* Add path and repo_id support for Diffusers Model Manager
Also fixes bugs
* Fix tooltip IT localization not working
* Add Version Number To WebUI
* Optimize Model Search
* Fix incorrect font on the Model Manager UI
* Fix image degradation on merge fixes - [Experimental]
This change should effectively fix a couple of things.
- Fix image degradation on subsequent merges of the canvas layers.
- Fix the slight transparent border that is left behind when filling the bounding box with a color.
- Fix the left over line of color when filling a bounding box with color.
So far there are no side effects for this. If any, please report.
* Add local model filtering for Diffusers / Checkpoints
* Go to home on modal close for the Add Modal UI
* Styling Fixes
* Model Manager Diffusers Localization Update
* Add Safe Tensor scanning to Model Manager
* Fix model edit form dispatching string values instead of numbers.
* Resolve VAE handling / edge cases for supplied repos
* defer injecting tokens for textual inversions until they're used for the first time
* squash a console warning
* implement model migration check
* add_model() overwrites previous config rather than merges
* fix model config file attribute merging
* fix precision handling in textual inversion script
* allow ckpt conversion script to work with safetensors .ckpts
Applied patch here:
https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47
* fix name "args" is not defined crash in textual_inversion_training
* fix a second NameError: name 'args' is not defined crash
* fix loading of the safety checker from the global cache dir
* add installation step to textual inversion frontend
- After a successful training run, the script will copy learned_embeds.bin
to a subfolder of the embeddings directory.
- User given the option to delete the logs and intermediate checkpoints
(which together use 7-8G of space)
- If textual inversion training fails, reports the error gracefully.
* don't crash out on incompatible embeddings
- put try: blocks around places where the system tries to load an embedding
which is incompatible with the currently loaded model
* add support for checkpoint resuming
* textual inversion preferences are saved and restored between sessions
- Preferences are stored in a file named text-inversion-training/preferences.conf
- Currently the resume-from-checkpoint option is not working correctly. Possible
bug in textual_inversion_training.py?
* copy learned_embeddings.bin into right location
* add front end for diffusers model merging
- Front end doesn't do anything yet!!!!
- Made change to model name parsing in CLI to support ability to have merged models
with the "+" character in their names.
* improve inpainting experience
- recommend ckpt version of inpainting-1.5 to user
- fix get_noise() bug in ckpt version of omnibus.py
* update environment*yml
* tweak instructions to install HuggingFace token
* bump version number
* enhance update scripts
- update scripts will now fetch new INITIAL_MODELS.yaml so that
configure_invokeai.py will know about the diffusers versions.
* enhance invoke.sh/invoke.bat launchers
- added configure_invokeai.py to menu
- menu defaults to browser-based invoke
* remove conda workflow (#2321)
* fix `token_ids has shape torch.Size([79]) - expected [77]`
* update CHANGELOG.md with 2.3.* info
- Add information on how formats have changed and the upgrade process.
- Add short bug list.
Co-authored-by: Damian Stewart <d@damianstewart.com>
Co-authored-by: Damian Stewart <null@damianstewart.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com>
Co-authored-by: mauwii <Mauwii@outlook.de>
Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com>
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com>
Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
< / s v g > ` . r e p l a c e A l l ( " b l a c k " , e ) , M B e = e = > { c o n s t { . . . t } = e , { m a s k C o l o r S t r i n g : n , s t a g e C o o r d i n a t e s : r , s t a g e D i m e n s i o n s : i , s t a g e S c a l e : o } = g e ( O B e ) , [ a , s ] = C . e x p o r t s . u s e S t a t e ( n u l l ) , [ l , u ] = C . e x p o r t s . u s e S t a t e ( 0 ) , f = C . e x p o r t s . u s e R e f ( n u l l ) , p = C . e x p o r t s . u s e C a l l b a c k ( ( ) = > { u ( l + 1 ) , s e t T i m e o u t ( p , 5 0 0 ) } , [ l ] ) ; r e t u r n C . e x p o r t s . u s e E f f e c t ( ( ) = > { i f ( a ) r e t u r n ; c o n s t m = n e w I m a g e ; m . o n l o a d = ( ) = > { s ( m ) } , m . s r c = k N ( n ) } , [ a , n ] ) , C . e x p o r t s . u s e E f f e c t ( ( ) = > { ! a | | ( a . s r c = k N ( n ) ) } , [ a , n ] ) , C . e x p o r t s . u s e E f f e c t ( ( ) = > { c o n s t m = s e t I n t e r v a l ( ( ) = > u ( v = > ( v + 1 ) % 5 ) , 5 0 ) ; r e t u r n ( ) = > c l e a r I n t e r v a l ( m ) } , [ ] ) , ! a | | ! g i . e x p o r t s . i s N u m b e r ( r . x ) | | ! g i . e x p o r t s . i s N u m b e r ( r . y ) | | ! g i . e x p o r t s . i s N u m b e r ( o ) | | ! g i . e x p o r t s . i s N u m b e r ( i . w i d t h ) | | ! g i . e x p o r t s . i s N u m b e r ( i . h e i g h t ) ? n u l l : y ( h c , { r e f : f , o f f s e t X : r . x / o , o f f s e t Y : r . y / o , h e i g h t : i . h e i g h t / o , w i d t h : i . w i d t h / o , f i l l P a t t e r n I m a g e : a , f i l l P a t t e r n O f f s e t Y : g i . e x p o r t s . i s N u m b e r ( l ) ? l : 0 , f i l l P a t t e r n R e p e a t : " r e p e a t " , f i l l P a t t e r n S c a l e : { x : 1 / o , y : 1 / o } , l i s t e n i n g : ! 0 , g l o b a l C o m p o s i t e O p e r a t i o n : " s o u r c e - i n " , . . . t } ) } , R B e = s t ( [ r n ] , e = > { c o n s t { i s M o v e S t a g e K e y H e l d : t , s t a g e S c a l e : n } = e ; r e t u r n { i s M o v e S t a g e K e y H e l d : t , s t a g e S c a l e : n } } , { m e m o i z e O p t i o n s : { r e s u l t E q u a l i t y C h e c k : F e . i s E q u a l } } ) , I B e = e = > { c o n s t t = B e ( ) , { i s M o v e S t a g e K e y H e l d : n , s t a g e S c a l e : r } = g e ( R B e ) ; r e t u r n C . e x p o r t s . u s e C a l l b a c k ( i = > { i f ( ! e . c u r r e n t | | n ) r e t u r n ; i . e v t . p r e v e n t D e f a u l t ( ) ; c o n s t o = e . c u r r e n t . g e t P o i n t e r P o s i t i o n ( ) ; i f ( ! o ) r e t u r n ; c o n s t a = { x : ( o . x - e . c u r r e n t . x ( ) ) / r , y : ( o . y - e . c u r r e n t . y ( ) ) / r } ; l e t s = i . e v t . d e l t a Y ; i . e v t . c t r l K e y & & ( s = - s ) ; c o n s t l = F e . c l a m p ( r * v w e * * s , y w e , b w e ) , u = { x : o . x - a . x * l , y : o . y - a . y * l } ; t ( G w e ( l ) ) , t ( S W ( u ) ) } , [ e , n , r , t ] ) } , t T = e = > { c o n s t t = e . g e t P o i n t e r P o s i t i o n ( ) , n = e . g e t A b s o l u t e T r a n s f o r m ( ) . c o p y ( ) ; i f ( ! t | | ! n ) r e t u r n ; c o n s t r = n . i n v e r t ( ) . p o i n t ( t ) ; r e t u r n { x : r . x , y : r . y } } , Q Y = ( ) = > { c o n s t e = B e ( ) , t = r l ( ) , n = b G ( ) ; r e t u r n { u p d a t e C o l o r U n d e r C u r s o r : ( ) = > { i f ( ! n | | ! t ) r e t u r n ; c o n s t r = n . g e t P o i n t e r P o s i t i o n ( ) ; i f ( ! r ) r e t u r n ; c o n s t i = F 0 . p i x e l R a t i o , [ o , a , s , l ] = t . g e t C o n t e x t ( ) . g e t I m a g e D a t a ( r . x * i , r . y * i , 1 , 1 ) . d a t a ; e ( z w e ( { r : o , g : a , b : s , a : l } ) ) } , c o m m i t C o l o r U n d e r C u r s o r : ( ) = > { e ( L w e ( ) ) } } } , D B e = s t ( [ P r , r n , T r ] , ( e , t , n ) = > { c o n s t { t o o l : r } = t ; r e t u r n { t o o l : r , a c t i v e T a b N a m e : e , i s S t a g i n g : n } } , { m e m o i z e O p t i o n s : { r e s u l t E q u a l i t y C h e c k : F e . i s E q u a l } } ) , N B e = e = > { c o n s t t = B e ( ) , { t o o l : n , i s S t a g i n g : r } = g e ( D B e ) , { c o m m i t C o l o r U n d e r C u r s o r : i } = Q Y ( ) ; r e t u r n C . e x p o r t s . u s e C a l l b a c k ( o = > { i f ( ! e . c u r r e n t ) r e t u r n ; i f ( e . c u r r e n t . c o n t a i n e r ( ) . f o c u s ( ) , n = = = " m o v e " | | r ) { t ( V 5 ( ! 0 ) ) ; r e t u r n } i f ( n = = = " c o l o r P i c k e r " ) { i ( ) ; r e t u r n } c o n s t a = t T ( e . c u r r e n t ) ; ! a | | ( o . e v t . p r e v e n t D e f a u l t ( ) , t ( d W ( ! 0 ) ) , t ( T w e ( [ a . x , a . y ] ) ) ) } , [ e , n , r , t , i ] ) } , B B e = s t ( [ P r , r n , T r ] , ( e , t , n ) = > { c o n s t { t o o l : r , i s D r a w i n g : i } = t ; r e t u r n { t o o l : r , i s D r a w i n g : i , a c t i v e T a b N a m e : e , i s S t a g i n g : n } } , { m e m o i z e O p t i o n s : { r e s u l t E q u a l i t y C h e c k : F e . i s E q u a l } } ) , $ B e = ( e , t ) = > { c o n s t n = B e ( ) , { t o o l : r , i s D r a w i n g : i , i s S t a g i n g : o } = g e ( B B e ) ; r e t u r n C . e x p o r t s . u s e C a l l b a c k ( ( ) = > { i f ( r = = = " m o v e " | | o ) { n ( V 5 ( ! 1 ) ) ; r e t u r n } i f ( ! t . c u r r e n t & & i & & e . c u r r e n t ) { c o n s t a = t T ( e . c u r r e n t ) ; i f ( ! a ) r e t u r n ; n ( l W ( [ a . x , a . y ] ) ) } e l s e t . c u r r e n t = ! 1 ; n ( d W ( ! 1 ) ) } , [ t , n , i , o , e , r ] ) } , F B e = s t ( [ P r , r n , T r ] , ( e , t , n ) = > { c o n s t { t o o l : r , i s D r a w i n g : i } = t ; r e t u r n { t o o l : r , i s D r a w i n g : i , a c t i v e T a b N a m e : e , i s S t a g i n g : n } } , { m e m o i z e O p t i o n s : { r e s u l t E q u a l i t y C h e c k : F e . i s E q u a l } } ) , z B e = ( e , t , n ) = > { c o n s t r = B e ( ) , { i s D r a w i n g : i , t o o l : o , i s S t a g i n g : a } = g e ( F B e ) , { u p d a t e C o l o r U n d e r C u r s o r : s } = Q Y ( ) ; r e t u r n C . e x p o r t s . u s e C a l l b a c k ( ( ) = > { i f ( ! e . c u r r e n t ) r e t u r n ; c o n s t l = t T ( e . c u r r e n t ) ; i f ( ! ! l ) { i f ( r ( j w e ( l ) ) , n . c u r r e n t = l , o = = = " c o l o r P i c k e r " ) { s ( ) ; r e t u r n } ! i | | o = = = " m o v e " | | a | | ( t . c u r r e n t = ! 0 , r ( l W ( [ l . x , l . y ] ) ) ) } } , [ t , r , i , a , n , e , o , s ] ) } , j B e = ( ) = > { c o n s t e = B e ( ) ; r e t u r n C . e x p o r t s . u s e C a l l b a c k ( ( ) = > { e ( M w e ( ) ) } , [ e ] ) } , H B e = s t ( [ r n , T r ] , ( e , t ) = > { c o n s t { t o o l : n , i s M o v i n g B o u n d i n g B o x : r } = e ; r e t u r n { t o o l : n , i s S t a g i n g : t , i s M o v i n g B o u n d i n g B o x : r } } , { m e m o i z e O p t i o n s : { r e s u l t E q u a l i t y C h e c k : F e . i s E q u a l } } ) , V B e = ( ) = > { c o n s t e = B e ( ) , { t o o l : t , i s S t a g i n g : n , i s M o v i n g B o u n d i n g B o x : r } = g e ( H B e ) ; r e t u r n { h a n d l e D r a g S t a r t : C . e x p o r t s . u s e C a l l b a c k ( ( ) = > { ! ( ( t = = = " m o v e " | | n ) & & ! r ) | | e ( V 5 ( ! 0 ) ) } , [ e , r , n , t ] ) , h a n d l e D r a g M o v e : C . e x p o r t s . u s e C a l l b a c k ( i = > { i f ( ! ( ( t = = = " m o v e " | | n ) & & ! r ) ) r e t u r n ; c o n s t o = { x : i . t a r g e t . x ( ) , y : i . t a r g e t . y ( ) } ; e ( S W ( o ) ) } , [ e , r , n , t ] ) , h a n d l e D r a g E n d : C . e x p o r t s . u s e C a l l b a c k ( ( ) = > { ! ( ( t = = = " m o v e " | | n ) & & ! r ) | | e ( V 5 ( ! 1 ) ) } , [ e , r , n , t ] ) } } ; v a r p h = C . e x p o r t s , U B e = f u n c t i o n ( t , n , r ) { c o n s t i = p h . u s e R e f ( " l o a d i n g " ) , o = p h . u s e R e f ( ) , [ a , s ] = p h . u s e S t a t e ( 0 ) , l = p h . u s e R e f ( ) , u = p h . u s e R e f ( ) , f = p h . u s e R e f ( ) ; r e t u r n ( l . c u r r e n t ! = = t | | u . c u r r e n t ! = = n | | f . c u r r e n t ! = = r ) & & ( i . c u r r e n t = "
In order to be iterable , non - array objects must have a [ Symbol . iterator ] ( ) method . ` )}function BFe(e){if(typeof Symbol<"u"&&e[Symbol.iterator]!=null||e["@@iterator"]!=null)return Array.from(e)}function $ Fe(e){if(Array.isArray(e))return Z8(e)}function RN(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter(function(i){return Object.getOwnPropertyDescriptor(e,i).enumerable})),n.push.apply(n,r)}return n}function IN(e){for(var t=1;t<arguments.length;t++){var n=arguments[t]!=null?arguments[t]:{};t%2?RN(Object(n),!0).forEach(function(r){dK(e,r,n[r])}):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):RN(Object(n)).forEach(function(r){Object.defineProperty(e,r,Object.getOwnPropertyDescriptor(n,r))})}return e}function dK(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function ry(e,t){return jFe(e)||zFe(e,t)||fK(e,t)||FFe()}function FFe(){throw new TypeError( ` Invalid attempt to destructure non - iterable instance .
In order to be iterable , non - array objects must have a [ Symbol . iterator ] ( ) method . ` )}function fK(e,t){if(!!e){if(typeof e=="string")return Z8(e,t);var n=Object.prototype.toString.call(e).slice(8,-1);if(n==="Object"&&e.constructor&&(n=e.constructor.name),n==="Map"||n==="Set")return Array.from(e);if(n==="Arguments"||/^(?:Ui|I)nt(?:8|16|32)(?:Clamped)?Array $ /.test(n))return Z8(e,t)}}function Z8(e,t){(t==null||t>e.length)&&(t=e.length);for(var n=0,r=new Array(t);n<t;n++)r[n]=e[n];return r}function zFe(e,t){var n=e==null?null:typeof Symbol<"u"&&e[Symbol.iterator]||e["@@iterator"];if(n!=null){var r=[],i=!0,o=!1,a,s;try{for(n=n.call(e);!(i=(a=n.next()).done)&&(r.push(a.value),!(t&&r.length===t));i=!0);}catch(l){o=!0,s=l}finally{try{!i&&n.return!=null&&n.return()}finally{if(o)throw s}}return r}}function jFe(e){if(Array.isArray(e))return e}var HFe="file-invalid-type",VFe="file-too-large",UFe="file-too-small",WFe="too-many-files",GFe=function(t){t=Array.isArray(t)&&t.length===1?t[0]:t;var n=Array.isArray(t)?"one of ".concat(t.join(", ")):t;return{code:HFe,message:"File type must be ".concat(n)}},DN=function(t){return{code:VFe,message:"File is larger than ".concat(t," ").concat(t===1?"byte":"bytes")}},NN=function(t){return{code:UFe,message:"File is smaller than ".concat(t," ").concat(t===1?"byte":"bytes")}},qFe={code:WFe,message:"Too many files"};function hK(e,t){var n=e.type==="application/x-moz-file"||DFe(e,t);return[n,n?null:GFe(t)]}function pK(e,t,n){if(kh(e.size))if(kh(t)&&kh(n)){if(e.size>n)return[!1,DN(n)];if(e.size<t)return[!1,NN(t)]}else{if(kh(t)&&e.size<t)return[!1,NN(t)];if(kh(n)&&e.size>n)return[!1,DN(n)]}return[!0,null]}function kh(e){return e!=null}function YFe(e){var t=e.files,n=e.accept,r=e.minSize,i=e.maxSize,o=e.multiple,a=e.maxFiles,s=e.validator;return!o&&t.length>1||o&&a>=1&&t.length>a?!1:t.every(function(l){var u=hK(l,n),f=ry(u,1),p=f[0],m=pK(l,r,i),v=ry(m,1),b=v[0],x=s?s(l):null;return p&&b&&!x})}function iS(e){return typeof e.isPropagationStopped=="function"?e.isPropagationStopped():typeof e.cancelBubble<"u"?e.cancelBubble:!1}function Ub(e){return e.dataTransfer?Array.prototype.some.call(e.dataTransfer.types,function(t){return t==="Files"||t==="application/x-moz-file"}):!!e.target&&!!e.target.files}function BN(e){e.preventDefault()}function KFe(e){return e.indexOf("MSIE")!==-1||e.indexOf("Trident/")!==-1}function XFe(e){return e.indexOf("Edge/")!==-1}function ZFe(){var e=arguments.length>0&&arguments[0]!==void 0?arguments[0]:window.navigator.userAgent;return KFe(e)||XFe(e)}function Nl(){for(var e=arguments.length,t=new Array(e),n=0;n<e;n++)t[n]=arguments[n];return function(r){for(var i=arguments.length,o=new Array(i>1?i-1:0),a=1;a<i;a++)o[a-1]=arguments[a];return t.some(function(s){return!iS(r)&&s&&s.apply(void 0,[r].concat(o)),iS(r)})}}function QFe(){return"showOpenFilePicker"in window}function JFe(e){if(kh(e)){var t=Object.entries(e).filter(function(n){var r=ry(n,2),i=r[0],o=r[1],a=!0;return gK(i)||(console.warn('Skipped "'.concat(i,'" because it is not a valid MIME type. Check https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types for a list of valid MIME types.')),a=!1),(!Array.isArray(o)||!o.every(mK))&&(console.warn('Skipped "'.concat(i,'" because an invalid file extension was provided.')),a=!1),a}).reduce(function(n,r){var i=ry(r,2),o=i[0],a=i[1];return IN(IN({},n),{},dK({},o,a))},{});return[{description:"Files",accept:t}]}return e}function eze(e){if(kh(e))return Object.entries(e).reduce(function(t,n){var r=ry(n,2),i=r[0],o=r[1];return[].concat(MN(t),[i],MN(o))},[]).filter(function(t){return gK(t)||mK(t)}).join(",")}function tze(e){return e instanceof DOMException&&(e.name==="AbortError"||e.code===e.ABORT_ERR)}function nze(e){return e instanceof DOMException&&(e.name==="SecurityError"||e.code===e.SECURITY_ERR)}function gK(e){return e==="audio/*"||e==="video/*"||e==="image/*"||e==="text/*"||/ \w + \/ [-+. \w ]+/g.test(e)}function mK(e){return/^.* \. [ \w ]+ $ /.test(e)}var rze=["children"],ize=["open"],oze=["refKey","role","onKeyDown","onFocus","onBlur","onClick","onDragEnter","o
In order to be iterable , non - array objects must have a [ Symbol . iterator ] ( ) method . ` )}function uze(e){if(typeof Symbol<"u"&&e[Symbol.iterator]!=null||e["@@iterator"]!=null)return Array.from(e)}function cze(e){if(Array.isArray(e))return Q8(e)}function VC(e,t){return hze(e)||fze(e,t)||vK(e,t)||dze()}function dze(){throw new TypeError( ` Invalid attempt to destructure non - iterable instance .
In order to be iterable , non - array objects must have a [ Symbol . iterator ] ( ) method . ` )}function vK(e,t){if(!!e){if(typeof e=="string")return Q8(e,t);var n=Object.prototype.toString.call(e).slice(8,-1);if(n==="Object"&&e.constructor&&(n=e.constructor.name),n==="Map"||n==="Set")return Array.from(e);if(n==="Arguments"||/^(?:Ui|I)nt(?:8|16|32)(?:Clamped)?Array $ /.test(n))return Q8(e,t)}}function Q8(e,t){(t==null||t>e.length)&&(t=e.length);for(var n=0,r=new Array(t);n<t;n++)r[n]=e[n];return r}function fze(e,t){var n=e==null?null:typeof Symbol<"u"&&e[Symbol.iterator]||e["@@iterator"];if(n!=null){var r=[],i=!0,o=!1,a,s;try{for(n=n.call(e);!(i=(a=n.next()).done)&&(r.push(a.value),!(t&&r.length===t));i=!0);}catch(l){o=!0,s=l}finally{try{!i&&n.return!=null&&n.return()}finally{if(o)throw s}}return r}}function hze(e){if(Array.isArray(e))return e}function $ N(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter(function(i){return Object.getOwnPropertyDescriptor(e,i).enumerable})),n.push.apply(n,r)}return n}function yr(e){for(var t=1;t<arguments.length;t++){var n=arguments[t]!=null?arguments[t]:{};t%2? $ N(Object(n),!0).forEach(function(r){J8(e,r,n[r])}):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)): $ N(Object(n)).forEach(function(r){Object.defineProperty(e,r,Object.getOwnPropertyDescriptor(n,r))})}return e}function J8(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function oS(e,t){if(e==null)return{};var n=pze(e,t),r,i;if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(i=0;i<o.length;i++)r=o[i],!(t.indexOf(r)>=0)&&(!Object.prototype.propertyIsEnumerable.call(e,r)||(n[r]=e[r]))}return n}function pze(e,t){if(e==null)return{};var n={},r=Object.keys(e),i,o;for(o=0;o<r.length;o++)i=r[o],!(t.indexOf(i)>=0)&&(n[i]=e[i]);return n}var oT=C.exports.forwardRef(function(e,t){var n=e.children,r=oS(e,rze),i=bK(r),o=i.open,a=oS(i,ize);return C.exports.useImperativeHandle(t,function(){return{open:o}},[o]),y(C.exports.Fragment,{children:n(yr(yr({},a),{},{open:o}))})});oT.displayName="Dropzone";var yK={disabled:!1,getFilesFromEvent:EFe,maxSize:1/0,minSize:0,multiple:!0,maxFiles:0,preventDropOnDocument:!0,noClick:!1,noKeyboard:!1,noDrag:!1,noDragEventsBubbling:!1,validator:null,useFsAccessApi:!0,autoFocus:!1};oT.defaultProps=yK;oT.propTypes={children:Fn.exports.func,accept:Fn.exports.objectOf(Fn.exports.arrayOf(Fn.exports.string)),multiple:Fn.exports.bool,preventDropOnDocument:Fn.exports.bool,noClick:Fn.exports.bool,noKeyboard:Fn.exports.bool,noDrag:Fn.exports.bool,noDragEventsBubbling:Fn.exports.bool,minSize:Fn.exports.number,maxSize:Fn.exports.number,maxFiles:Fn.exports.number,disabled:Fn.exports.bool,getFilesFromEvent:Fn.exports.func,onFileDialogCancel:Fn.exports.func,onFileDialogOpen:Fn.exports.func,useFsAccessApi:Fn.exports.bool,autoFocus:Fn.exports.bool,onDragEnter:Fn.exports.func,onDragLeave:Fn.exports.func,onDragOver:Fn.exports.func,onDrop:Fn.exports.func,onDropAccepted:Fn.exports.func,onDropRejected:Fn.exports.func,onError:Fn.exports.func,validator:Fn.exports.func};var e_={isFocused:!1,isFileDialogActive:!1,isDragActive:!1,isDragAccept:!1,isDragReject:!1,acceptedFiles:[],fileRejections:[]};function bK(){var e=arguments.length>0&&arguments[0]!==void 0?arguments[0]:{},t=yr(yr({},yK),e),n=t.accept,r=t.disabled,i=t.getFilesFromEvent,o=t.maxSize,a=t.minSize,s=t.multiple,l=t.maxFiles,u=t.onDragEnter,f=t.onDragLeave,p=t.onDragOver,m=t.onDrop,v=t.onDropAccepted,b=t.onDropRejected,x=t.onFileDialogCancel,E=t.onFileDialogOpen,k=t.useFsAccessApi,_=t.autoFocus,T=t.preventDropOnDocument,A=t.noClick,R=t.noKeyboard,I=t.noDrag,D=t.noDragEventsBubbling,B=t.onError,z=t.validator,j=C.exports.useMemo(function(){return eze(n)},[n]),K=C.exports.useMemo(function(){return JFe(n)},[n]),te=C.exports.useMemo(function(){return typeof E=="function"?E:FN},[E]),G=C.exports.useMemo(function(){return typeof x=="function"?x:FN},[x]), $ =C.exports.useRef(null),V=C.exports.useRef(null),J=C.export
` +I.message,"");i({title:o("toast:uploadFailed"),description:A,status:"error",isClosable:!0})},[o,i]),f=C.exports.useCallback(async T=>{n(zN({imageFile:T}))},[n]),p=C.exports.useCallback((T,A)=>{A.forEach(R=>{u(R)}),T.forEach(R=>{f(R)})},[f,u]),{getRootProps:m,getInputProps:v,isDragAccept:b,isDragReject:x,isDragActive:E,open:k}=bK({accept:{"image/png":[".png"],"image/jpeg":[".jpg",".jpeg",".png"]},noClick:!0,onDrop:p,onDragOver:()=>s(!0),maxFiles:1});l(k),C.exports.useEffect(()=>{const T=A=>{var B;const R=(B=A.clipboardData)==null?void 0:B.items;if(!R)return;const I=[];for(const z of R)z.kind==="file"&&["image/png","image/jpg"].includes(z.type)&&I.push(z);if(!I.length)return;if(A.stopImmediatePropagation(),I.length>1){i({description:o("toast:uploadFailedMultipleImagesDesc"),status:"error",isClosable:!0});return}const D=I[0].getAsFile();if(!D){i({description:o("toast:uploadFailedUnableToLoadDesc"),status:"error",isClosable:!0});return}n(zN({imageFile:D}))};return document.addEventListener("paste",T),()=>{document.removeEventListener("paste",T)}},[o,n,i,r]);const _=["img2img","unifiedCanvas"].includes(r)? ` to $ { rs [ r ] . tooltip } ` :"";return y(jP.Provider,{value:k,children:q("div",{...m({style:{}}),onKeyDown:T=>{T.key},children:[y("input",{...v()}),t,E&&a&&y(mze,{isDragAccept:b,isDragReject:x,overlaySecondaryText:_,setIsHandlingUpload:s})]})})},yze=()=>{const e=Be(),t=ge(K8e),n=Ty();C.exports.useEffect(()=>{t.forEach(r=>{n(r)}),t.length>0&&e(pwe())},[e,n,t])},SK=st([e=>e.options,e=>e.gallery,Pr],(e,t,n)=>{const{shouldPinOptionsPanel:r,shouldShowOptionsPanel:i,shouldHoldOptionsPanelOpen:o,shouldUseCanvasBetaLayout:a}=e,{shouldShowGallery:s,shouldPinGallery:l,shouldHoldGalleryOpen:u}=t,f=a&&n==="unifiedCanvas",p=!f&&!(i||o&&!r)&&["txt2img","img2img","unifiedCanvas"].includes(n),m=!(s||u&&!l)&&["txt2img","img2img","unifiedCanvas"].includes(n);return{shouldPinOptionsPanel:r,shouldShowProcessButtons:!f&&(!r||!i),shouldShowOptionsPanelButton:p,shouldShowOptionsPanel:i,shouldShowGallery:s,shouldPinGallery:l,shouldShowGalleryButton:m}},{memoizeOptions:{resultEqualityCheck:Fe.isEqual}}),bze=()=>{const e=Be(),{shouldShowOptionsPanel:t,shouldShowOptionsPanelButton:n,shouldShowProcessButtons:r,shouldPinOptionsPanel:i,shouldShowGallery:o,shouldPinGallery:a}=ge(SK),s=()=>{e(Ju(!0)),i&&setTimeout(()=>e(vi(!0)),400)};return Ze("f",()=>{o||t?(e(Ju(!1)),e(Vd(!1))):(e(Ju(!0)),e(Vd(!0))),(a||i)&&setTimeout(()=>e(vi(!0)),400)},[o,t]),n?q("div",{className:"show-hide-button-options",children:[y(Qe,{tooltip:"Show Options Panel (O)",tooltipProps:{placement:"top"},"aria-label":"Show Options Panel",onClick:s,children:y(dP,{})}),r&&q(fn,{children:[y(DP,{iconButton:!0}),y(NP,{})]})]}):null},Sze=()=>{const e=Be(),{shouldShowGallery:t,shouldShowGalleryButton:n,shouldPinGallery:r,shouldShowOptionsPanel:i,shouldPinOptionsPanel:o}=ge(SK),a=()=>{e(Vd(!0)),r&&e(vi(!0))};return Ze("f",()=>{t||i?(e(Ju(!1)),e(Vd(!1))):(e(Ju(!0)),e(Vd(!0))),(r||o)&&setTimeout(()=>e(vi(!0)),400)},[t,i]),n?y(Qe,{tooltip:"Show Gallery (G)",tooltipProps:{placement:"top"},"aria-label":"Show Gallery",styleClass:"floating-show-hide-button right show-hide-button-gallery",onClick:a,children:y(wq,{})}):null};dOe();const xze=()=>(yze(),q("div",{className:"App",children:[q(vze,{children:[y(m8e,{}),q("div",{className:"app-content",children:[y(bLe,{}),y(wFe,{})]}),y("div",{className:"app-console",children:y(cOe,{})})]}),y(bze,{}),y(Sze,{})]}));const wze=SF({key:"invokeai-style-cache",prepend:!0});WC.createRoot(document.getElementById("root")).render(y(oe.StrictMode,{children:y(s8e,{store:xG,children:y(OG,{loading:y(ZI,{}),persistor:MG,children:y(Lie,{value:wze,children:y(_4e,{children:y(oe.Suspense,{fallback:y(ZI,{}),children:y(xze,{})})})})})})}));export{_ze as __vite_legacy_guard};