Mirrored_Repos/InvokeAI
1
0
Fork 0
mirror of https://github.com/invoke-ai/InvokeAI synced 2024-08-30 20:32:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
43ebd68313
InvokeAI/invokeai/configs/stable-diffusion/v2-inference-v.yaml

69 lines
1.9 KiB
YAML
Raw Normal View History

use 🧨diffusers model (#1583) * initial commit of DiffusionPipeline class * spike: proof of concept using diffusers for txt2img * doc: type hints for Generator * refactor(model_cache): factor out load_ckpt * model_cache: add ability to load a diffusers model pipeline and update associated things in Generate & Generator to not instantly fail when that happens * model_cache: fix model default image dimensions * txt2img: support switching diffusers schedulers * diffusers: let the scheduler do its scaling of the initial latents Remove IPNDM scheduler; it is not behaving. * web server: update image_progress callback for diffusers data * diffusers: restore prompt weighting feature * diffusers: fix set-sampler error following model switch * diffusers: use InvokeAIDiffuserComponent for conditioning * cross_attention_control: stub (no-op) implementations for diffusers * model_cache: let offload_model work with DiffusionPipeline, sorta. * models.yaml.example: add diffusers-format model, set as default * test-invoke-conda: use diffusers-format model test-invoke-conda: put huggingface-token where the library can use it * environment-mac: upgrade to diffusers 0.7 (from 0.6) this was already done for linux; mac must have been lost in the merge. * preload_models: explicitly load diffusers models In non-interactive mode too, as long as you're logged in. * fix(model_cache): don't check `model.config` in diffusers format clean-up from recent merge. * diffusers integration: support img2img * dev: upgrade to diffusers 0.8 (from 0.7.1) We get to remove some code by using methods that were factored out in the base class. * refactor: remove backported img2img.get_timesteps now that we can use it directly from diffusers 0.8.1 * ci: use diffusers model * dev: upgrade to diffusers 0.9 (from 0.8.1) * lint: correct annotations for Python 3.9. * lint: correct AttributeError.name reference for Python 3.9. * CI: prefer diffusers-1.4 because it no longer requires a token The RunwayML models still do. * build: there's yet another place to update requirements? * configure: try to download models even without token Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.) * configure: add troubleshooting info for config-not-found * fix(configure): prepend root to config path * fix(configure): remove second `default: true` from models example * CI: simplify test-on-push logic now that we don't need secrets The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks. * create an embedding_manager for diffusers * internal: avoid importing diffusers DummyObject see https://github.com/huggingface/diffusers/issues/1479 * fix "config attributes…not expected" diffusers warnings. * fix deprecated scheduler construction * work around an apparent MPS torch bug that causes conditioning to have no effect * 🚧 post-rebase repair * preliminary support for outpainting (no masking yet) * monkey-patch diffusers.attention and use Invoke lowvram code * add always_use_cpu arg to bypass MPS * add cross-attention control support to diffusers (fails on MPS) For unknown reasons MPS produces garbage output with .swap(). Use --always_use_cpu arg to invoke.py for now to test this code on MPS. * diffusers support for the inpainting model * fix debug_image to not crash with non-RGB images. * inpainting for the normal model [WIP] This seems to be performing well until the LAST STEP, at which point it dissolves to confetti. * fix off-by-one bug in cross-attention-control (#1774) prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness). based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly. * refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary * inpainting for the normal model. I think it works this time. * diffusers: reset num_vectors_per_token sync with 44a00555718f1df173c60da0ed646cf700e29537 * diffusers: txt2img2img (hires_fix) with so much slicing and dicing of pipeline methods to stitch them together * refactor(diffusers): reduce some code duplication amongst the different tasks * fixup! refactor(diffusers): reduce some code duplication amongst the different tasks * diffusers: enable DPMSolver++ scheduler * diffusers: upgrade to diffusers 0.10, add Heun scheduler * diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers * CI: default to diffusers-1.5 now that runwayml token requirement is gone * diffusers: update to 0.10 (and transformers to 4.25) * diffusers: use xformers when available diffusers no longer auto-enables this as of 0.10.2. * diffusers: make masked img2img behave better with multi-step schedulers re-randomizing the noise each step was confusing them. * diffusers: work more better with more models. fixed relative path problem with local models. fixed models on hub not always having a `fp16` branch. * diffusers: stopgap fix for attention_maps_callback crash after recent merge * fixup import merge conflicts correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b * test: add tests/inpainting inputs for masked img2img * diffusers(AddsMaskedGuidance): partial fix for k-schedulers Prevents them from crashing, but results are still hot garbage. * fix --safety_checker arg parsing and add note to diffusers loader about where safety checker gets called * generate: fix import error * CI: don't try to read the old init location * diffusers: support loading an alternate VAE * CI: remove sh-syntax if-statement so it doesn't crash powershell * CI: fold strings in yaml because backslash is not line-continuation in powershell * attention maps callback stuff for diffusers * build: fix syntax error in environment-mac * diffusers: add INITIAL_MODELS with diffusers-compatible repos * re-enable the embedding manager; closes #1778 * Squashed commit of the following: commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:43:07 2022 +0100 import new load handling from EmbeddingManager and cleanup commit c4abe91a5ba0d415b45bf734068385668b7a66e6 Merge: 032e856e 1efc6397 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:09:53 2022 +0100 Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager commit 032e856eefb3bbc39534f5daafd25764bcfcef8b Merge: 8b4f0fe9 bc515e24 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:08:01 2022 +0100 Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager commit 1efc6397fc6e61c1aff4b0258b93089d61de5955 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:04:28 2022 +0100 cleanup and add performance notes commit e400f804ac471a0ca2ba432fd658778b20c7bdab Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 14:45:07 2022 +0100 fix bug and update unit tests commit deb9ae0ae1016750e93ce8275734061f7285a231 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 14:28:29 2022 +0100 textual inversion manager seems to work commit 162e02505dec777e91a983c4d0fb52e950d25ff0 Merge: cbad4583 12769b3d Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:58:03 2022 +0100 Merge branch 'main' into feature_textual_inversion_mgr commit cbad45836c6aace6871a90f2621a953f49433131 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:54:10 2022 +0100 use position embeddings commit 070344c69b0e0db340a183857d0a787b348681d3 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:53:47 2022 +0100 Don't crash CLI on exceptions commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:11:55 2022 +0100 add missing position_embeddings commit 12769b3d3562ef71e0f54946b532ad077e10043c Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 13:33:25 2022 +0100 debugging why it don't work commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 13:21:33 2022 +0100 debugging why it don't work commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1 Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 12:48:38 2022 +0100 use TextualInversionManager in place of embeddings (wip, doesn't work) commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 12:48:38 2022 +0100 use TextualInversionManager in place of embeddings (wip, doesn't work) commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf Merge: 6e4dad60 023df37e Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:37:31 2022 +0100 Merge branch 'feature_textual_inversion_mgr' into dev/diffusers commit 023df37efffa67434f77def7fc3c9dfb29f699fd Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:36:54 2022 +0100 cleanup commit 05fac594eaf79d0058e3c48deee93df603f136c2 Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:07:49 2022 +0100 tweak error checking commit 009f32ed39a7280997c3ffab112adadee0b44279 Author: damian <null@damianstewart.com> Date: Thu Dec 15 21:29:47 2022 +0100 unit tests passing for embeddings with vector length >1 commit beb1b08d9a98112ed2fe073580568e1a18698da3 Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 13:39:09 2022 +0100 more explicit equality tests when overwriting commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9 Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 13:30:13 2022 +0100 wip textual inversion manager (unit tests passing for 1v embedding overwriting) commit 417c2b57d90924a839616bfb66804faab8039e4c Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 12:30:55 2022 +0100 wip textual inversion manager (unit tests passing for base stuff + padding) commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 10:57:57 2022 +0100 wip new TextualInversionManager * stop using WeightedFrozenCLIPEmbedder * store diffusion models locally - configure_invokeai.py reconfigured to store diffusion models rather than CompVis models - hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id - models.yaml does **NOT** use path, just repo_id - "repo_name" changed to "repo_id" to following hugging face conventions - Models are loaded with full precision pending further work. * allow non-local files during development * path takes priority over repo_id * MVP for model_cache and configure_invokeai - Feature complete (almost) - configure_invokeai.py downloads both .ckpt and diffuser models, along with their VAEs. Both types of download are controlled by a unified INITIAL_MODELS.yaml file. - model_cache can load both type of model and switches back and forth in CPU. No memory leaks detected TO DO: 1. I have not yet turned on the LocalOnly flag for diffuser models, so the code will check the Hugging Face repo for updates before using the locally cached models. This will break firewalled systems. I am thinking of putting in a global check for internet connectivity at startup time and setting the LocalOnly flag based on this. It would be good to check updates if there is connectivity. 2. I have not gone completely through INITIAL_MODELS.yaml to check which models are available as diffusers and which are not. So models like PaperCut and VoxelArt may not load properly. The runway and stability models are checked, as well as the Trinart models. 3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE: 1. When loading a .ckpt file there are lots of messages like this: Warning! ldm.modules.attention.CrossAttention is no longer being maintained. Please use InvokeAICrossAttention instead. I'm not sure how to address this. 2. The ckpt models ***don't actually run*** due to the lack of special-case support for them in the generator objects. For example, here's the hard crash you get when you run txt2img against the legacy waifu-diffusion-1.3 model: ``` >> An error occurred: Traceback (most recent call last): File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main main_loop(gen, opt) File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop gen.prompt2image( File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image results = generator.generate( File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate image = make_image(x_T) File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image pipeline_output = pipeline.image_from_embeddings( File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__ raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings' ``` 3. The inpainting diffusion model isn't working. Here's the output of "banana sushi" when inpainting-1.5 is loaded: ``` Traceback (most recent call last): File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image results = generator.generate( File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate image = make_image(x_T) File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image pipeline_output = pipeline.image_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings result_latents, result_attention_map_saver = self.latents_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings result: PipelineIntermediateState = infer_latents_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__ for result in self.generator_method(*args, **kwargs): File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings step_output = self.step(batched_t, latents, guidance_scale, File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs) File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step pred_original_sample = sample - sigma * model_output RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1 ``` * proper support for float32/float16 - configure script now correctly detects user's preference for fp16/32 and downloads the correct diffuser version. If fp16 version not available, falls back to fp32 version. - misc code cleanup and simplification in model_cache * add on-the-fly conversion of .ckpt to diffusers models 1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py. 2. A new !optimize command has been added to the CLI. Should be ported to Web GUI. User experience on the CLI is this: ``` invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model. This operation will take 30-60s to complete. Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 Writing new config file entry for sd-v1-4... >> New configuration: sd-v1-4: description: Optimized version of sd-v1-4 format: diffusers path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 OK to import [n]? y >> Verifying that new model loads... >> Current VRAM usage: 2.60G >> Offloading stable-diffusion-2.1 to CPU >> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 | Using faster float16 precision You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \ license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\ disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 . | training width x height = (512 x 512) >> Model loaded in 3.48s >> Max VRAM used to load the model: 2.17G >> Current VRAM usage:2.17G >> Textual inversions available: >> Setting Sampler to k_lms (LMSDiscreteScheduler) Keep model loaded? [y] ``` * add parallel set of generator files for ckpt legacy generation * generation using legacy ckpt models now working * diffusers: fix missing attention_maps_callback fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36 * associate legacy CrossAttention with .ckpt models * enable autoconvert New --autoconvert CLI option will scan a designated directory for new .ckpt files, convert them into diffuser models, and import them into models.yaml. Works like this: invoke.py --autoconvert /path/to/weights/directory In ModelCache added two new methods: autoconvert_weights(config_path, weights_directory_path, models_directory_path) convert_and_import(ckpt_path, diffuser_path) * diffusers: update to diffusers 0.11 (from 0.10.2) * fix vae loading & width/height calculation * refactor: encapsulate these conditioning data into one container * diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function * add support for safetensors and accelerate * set local_files_only when internet unreachable * diffusers: fix error-handling path when model repo has no fp16 branch * fix generatorinpaint error Fixes : "ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint' https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318 * quench diffuser safety-checker warning * diffusers: support stochastic DDIM eta parameter * fix conda env creation on macos * fix cross-attention with diffusers 0.11 * diffusers: the VAE needs to be tiling as well as the U-Net * diffusers: comment on subfolders * diffusers: embiggen! * diffusers: make model_cache.list_models serializable * diffusers(inpaint): restore scaling functionality * fix requirements clash between numba and numpy 1.24 * diffusers: allow inpainting model to do non-inpainting tasks * start expanding model_cache functionality * add import_ckpt_model() and import_diffuser_model() methods to model_manager - in addition, model_cache.py is now renamed to model_manager.py * allow "recommended" flag to be optional in INITIAL_MODELS.yaml * configure_invokeai now downloads VAE diffusers in advance * rename ModelCache to ModelManager * remove support for `repo_name` in models.yaml * check for and refuse to load embeddings trained on incompatible models * models.yaml.example: s/repo_name/repo_id and remove extra INITIAL_MODELS now that the main one has diffusers models in it. * add MVP textual inversion script * refactor(InvokeAIDiffuserComponent): factor out _combine() * InvokeAIDiffuserComponent: implement threshold * InvokeAIDiffuserComponent: diagnostic logs for threshold ...this does not look right * add a curses-based frontend to textual inversion - not quite working yet - requires npyscreen installed - on windows will also have the windows-curses requirement, but not added to requirements yet * add curses-based interface for textual inversion * fix crash in convert_and_import() - This corrects a "local variable referenced before assignment" error in model_manager.convert_and_import() * potential workaround for no 'state_dict' key error - As reported in https://github.com/huggingface/diffusers/issues/1876 * create TI output dir if needed * Update environment-lin-cuda.yml (#2159) Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~ * diffusers: update sampler-to-scheduler mapping based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672 * improve user exp for ckt to diffusers conversion - !optimize_models command now operates on an existing ckpt file entry in models.yaml - replaces existing entry, rather than adding a new one - offers to delete the ckpt file after conversion * web: adapt progress callback to deal with old generator or new diffusers pipeline * clean-up model_manager code - add_model() verified to work for .ckpt local paths, .ckpt remote URLs, diffusers local paths, and diffusers repo_ids - convert_and_import() verified to work for local and remove .ckpt files * handle edge cases for import_model() and convert_model() * add support for safetensor .ckpt files * fix name error * code cleanup with pyflake * improve model setting behavior - If the user enters an invalid model name at startup time, will not try to load it, warn, and use default model - CLI UI enhancement: include currently active model in the command line prompt. * update test-invoke-pip.yml - fix model cache path to point to runwayml/stable-diffusion-v1-5 - remove `skip-sd-weights` from configure_invokeai.py args * exclude dev/diffusers from "fail for draft PRs" * disable "fail on PR jobs" * re-add `--skip-sd-weights` since no space * update workflow environments - include `INVOKE_MODEL_RECONFIGURE: '--yes'` * clean up model load failure handling - Allow CLI to run even when no model is defined or loadable. - Inhibit stack trace when model load fails - only show last error - Give user *option* to run configure_invokeai.py when no models successfully load. - Restart invokeai after reconfiguration. * further edge-case handling 1) only one model in models.yaml file, and that model is broken 2) no models in models.yaml 3) models.yaml doesn't exist at all * fix incorrect model status listing - "cached" was not being returned from list_models() - normalize handling of exceptions during model loading: - Passing an invalid model name to generate.set_model() will return a KeyError - All other exceptions are returned as the appropriate Exception * CI: do download weights (if not already cached) * diffusers: fix scheduler loading in offline mode * CI: fix model name (no longer has `diffusers-` prefix) * Update txt2img2img.py (#2256) * fixes to share models with HuggingFace cache system - If HF_HOME environment variable is defined, then all huggingface models are stored in that directory following the standard conventions. - For seamless interoperability, set HF_HOME to ~/.cache/huggingface - If HF_HOME not defined, then models are stored in ~/invokeai/models. This is equivalent to setting HF_HOME to ~/invokeai/models A future commit will add a migration mechanism so that this change doesn't break previous installs. * feat - make model storage compatible with hugging face caching system This commit alters the InvokeAI model directory to be compatible with hugging face, making it easier to share diffusers (and other models) across different programs. - If the HF_HOME environment variable is not set, then models are cached in ~/invokeai/models in a format that is identical to the HuggingFace cache. - If HF_HOME is set, then models are cached wherever HF_HOME points. - To enable sharing with other HuggingFace library clients, set HF_HOME to ~/.cache/huggingface to set the default cache location or to ~/invokeai/models to have huggingface cache inside InvokeAI. * fixes to share models with HuggingFace cache system - If HF_HOME environment variable is defined, then all huggingface models are stored in that directory following the standard conventions. - For seamless interoperability, set HF_HOME to ~/.cache/huggingface - If HF_HOME not defined, then models are stored in ~/invokeai/models. This is equivalent to setting HF_HOME to ~/invokeai/models A future commit will add a migration mechanism so that this change doesn't break previous installs. * fix error "no attribute CkptInpaint" * model_manager.list_models() returns entire model config stanza+status * Initial Draft - Model Manager Diffusers * added hash function to diffusers * implement sha256 hashes on diffusers models * Add Model Manager Support for Diffusers * fix various problems with model manager - in cli import functions, fix not enough values to unpack from _get_name_and_desc() - fix crash when using old-style vae: value with new-style diffuser * rebuild frontend * fix dictconfig-not-serializable issue * fix NoneType' object is not subscriptable crash in model_manager * fix "str has no attribute get" error in model_manager list_models() * Add path and repo_id support for Diffusers Model Manager Also fixes bugs * Fix tooltip IT localization not working * Add Version Number To WebUI * Optimize Model Search * Fix incorrect font on the Model Manager UI * Fix image degradation on merge fixes - [Experimental] This change should effectively fix a couple of things. - Fix image degradation on subsequent merges of the canvas layers. - Fix the slight transparent border that is left behind when filling the bounding box with a color. - Fix the left over line of color when filling a bounding box with color. So far there are no side effects for this. If any, please report. * Add local model filtering for Diffusers / Checkpoints * Go to home on modal close for the Add Modal UI * Styling Fixes * Model Manager Diffusers Localization Update * Add Safe Tensor scanning to Model Manager * Fix model edit form dispatching string values instead of numbers. * Resolve VAE handling / edge cases for supplied repos * defer injecting tokens for textual inversions until they're used for the first time * squash a console warning * implement model migration check * add_model() overwrites previous config rather than merges * fix model config file attribute merging * fix precision handling in textual inversion script * allow ckpt conversion script to work with safetensors .ckpts Applied patch here: https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47 * fix name "args" is not defined crash in textual_inversion_training * fix a second NameError: name 'args' is not defined crash * fix loading of the safety checker from the global cache dir * add installation step to textual inversion frontend - After a successful training run, the script will copy learned_embeds.bin to a subfolder of the embeddings directory. - User given the option to delete the logs and intermediate checkpoints (which together use 7-8G of space) - If textual inversion training fails, reports the error gracefully. * don't crash out on incompatible embeddings - put try: blocks around places where the system tries to load an embedding which is incompatible with the currently loaded model * add support for checkpoint resuming * textual inversion preferences are saved and restored between sessions - Preferences are stored in a file named text-inversion-training/preferences.conf - Currently the resume-from-checkpoint option is not working correctly. Possible bug in textual_inversion_training.py? * copy learned_embeddings.bin into right location * add front end for diffusers model merging - Front end doesn't do anything yet!!!! - Made change to model name parsing in CLI to support ability to have merged models with the "+" character in their names. * improve inpainting experience - recommend ckpt version of inpainting-1.5 to user - fix get_noise() bug in ckpt version of omnibus.py * update environment*yml * tweak instructions to install HuggingFace token * bump version number * enhance update scripts - update scripts will now fetch new INITIAL_MODELS.yaml so that configure_invokeai.py will know about the diffusers versions. * enhance invoke.sh/invoke.bat launchers - added configure_invokeai.py to menu - menu defaults to browser-based invoke * remove conda workflow (#2321) * fix `token_ids has shape torch.Size([79]) - expected [77]` * update CHANGELOG.md with 2.3.* info - Add information on how formats have changed and the upgrade process. - Add short bug list. Co-authored-by: Damian Stewart <d@damianstewart.com> Co-authored-by: Damian Stewart <null@damianstewart.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com> Co-authored-by: mauwii <Mauwii@outlook.de> Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com> Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com> Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com> Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
model:
base_learning_rate: 1.0e-4
support both epsilon and v-prediction v2 inference There are actually two Stable Diffusion v2 legacy checkpoint configurations: 1) "epsilon" prediction type for Stable Diffusion v2 Base 2) "v-prediction" type for Stable Diffusion v2-768 This commit adds the configuration file needed for epsilon prediction type models as well as the UI that prompts the user to select the appropriate configuration file when the code can't do so automatically.
2023-03-06 03:51:40 +00:00
target: invokeai.backend.stable_diffusion.diffusion.ddpm.LatentDiffusion
use 🧨diffusers model (#1583) * initial commit of DiffusionPipeline class * spike: proof of concept using diffusers for txt2img * doc: type hints for Generator * refactor(model_cache): factor out load_ckpt * model_cache: add ability to load a diffusers model pipeline and update associated things in Generate & Generator to not instantly fail when that happens * model_cache: fix model default image dimensions * txt2img: support switching diffusers schedulers * diffusers: let the scheduler do its scaling of the initial latents Remove IPNDM scheduler; it is not behaving. * web server: update image_progress callback for diffusers data * diffusers: restore prompt weighting feature * diffusers: fix set-sampler error following model switch * diffusers: use InvokeAIDiffuserComponent for conditioning * cross_attention_control: stub (no-op) implementations for diffusers * model_cache: let offload_model work with DiffusionPipeline, sorta. * models.yaml.example: add diffusers-format model, set as default * test-invoke-conda: use diffusers-format model test-invoke-conda: put huggingface-token where the library can use it * environment-mac: upgrade to diffusers 0.7 (from 0.6) this was already done for linux; mac must have been lost in the merge. * preload_models: explicitly load diffusers models In non-interactive mode too, as long as you're logged in. * fix(model_cache): don't check `model.config` in diffusers format clean-up from recent merge. * diffusers integration: support img2img * dev: upgrade to diffusers 0.8 (from 0.7.1) We get to remove some code by using methods that were factored out in the base class. * refactor: remove backported img2img.get_timesteps now that we can use it directly from diffusers 0.8.1 * ci: use diffusers model * dev: upgrade to diffusers 0.9 (from 0.8.1) * lint: correct annotations for Python 3.9. * lint: correct AttributeError.name reference for Python 3.9. * CI: prefer diffusers-1.4 because it no longer requires a token The RunwayML models still do. * build: there's yet another place to update requirements? * configure: try to download models even without token Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.) * configure: add troubleshooting info for config-not-found * fix(configure): prepend root to config path * fix(configure): remove second `default: true` from models example * CI: simplify test-on-push logic now that we don't need secrets The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks. * create an embedding_manager for diffusers * internal: avoid importing diffusers DummyObject see https://github.com/huggingface/diffusers/issues/1479 * fix "config attributes…not expected" diffusers warnings. * fix deprecated scheduler construction * work around an apparent MPS torch bug that causes conditioning to have no effect * 🚧 post-rebase repair * preliminary support for outpainting (no masking yet) * monkey-patch diffusers.attention and use Invoke lowvram code * add always_use_cpu arg to bypass MPS * add cross-attention control support to diffusers (fails on MPS) For unknown reasons MPS produces garbage output with .swap(). Use --always_use_cpu arg to invoke.py for now to test this code on MPS. * diffusers support for the inpainting model * fix debug_image to not crash with non-RGB images. * inpainting for the normal model [WIP] This seems to be performing well until the LAST STEP, at which point it dissolves to confetti. * fix off-by-one bug in cross-attention-control (#1774) prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness). based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly. * refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary * inpainting for the normal model. I think it works this time. * diffusers: reset num_vectors_per_token sync with 44a00555718f1df173c60da0ed646cf700e29537 * diffusers: txt2img2img (hires_fix) with so much slicing and dicing of pipeline methods to stitch them together * refactor(diffusers): reduce some code duplication amongst the different tasks * fixup! refactor(diffusers): reduce some code duplication amongst the different tasks * diffusers: enable DPMSolver++ scheduler * diffusers: upgrade to diffusers 0.10, add Heun scheduler * diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers * CI: default to diffusers-1.5 now that runwayml token requirement is gone * diffusers: update to 0.10 (and transformers to 4.25) * diffusers: use xformers when available diffusers no longer auto-enables this as of 0.10.2. * diffusers: make masked img2img behave better with multi-step schedulers re-randomizing the noise each step was confusing them. * diffusers: work more better with more models. fixed relative path problem with local models. fixed models on hub not always having a `fp16` branch. * diffusers: stopgap fix for attention_maps_callback crash after recent merge * fixup import merge conflicts correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b * test: add tests/inpainting inputs for masked img2img * diffusers(AddsMaskedGuidance): partial fix for k-schedulers Prevents them from crashing, but results are still hot garbage. * fix --safety_checker arg parsing and add note to diffusers loader about where safety checker gets called * generate: fix import error * CI: don't try to read the old init location * diffusers: support loading an alternate VAE * CI: remove sh-syntax if-statement so it doesn't crash powershell * CI: fold strings in yaml because backslash is not line-continuation in powershell * attention maps callback stuff for diffusers * build: fix syntax error in environment-mac * diffusers: add INITIAL_MODELS with diffusers-compatible repos * re-enable the embedding manager; closes #1778 * Squashed commit of the following: commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:43:07 2022 +0100 import new load handling from EmbeddingManager and cleanup commit c4abe91a5ba0d415b45bf734068385668b7a66e6 Merge: 032e856e 1efc6397 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:09:53 2022 +0100 Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager commit 032e856eefb3bbc39534f5daafd25764bcfcef8b Merge: 8b4f0fe9 bc515e24 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:08:01 2022 +0100 Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager commit 1efc6397fc6e61c1aff4b0258b93089d61de5955 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:04:28 2022 +0100 cleanup and add performance notes commit e400f804ac471a0ca2ba432fd658778b20c7bdab Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 14:45:07 2022 +0100 fix bug and update unit tests commit deb9ae0ae1016750e93ce8275734061f7285a231 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 14:28:29 2022 +0100 textual inversion manager seems to work commit 162e02505dec777e91a983c4d0fb52e950d25ff0 Merge: cbad4583 12769b3d Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:58:03 2022 +0100 Merge branch 'main' into feature_textual_inversion_mgr commit cbad45836c6aace6871a90f2621a953f49433131 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:54:10 2022 +0100 use position embeddings commit 070344c69b0e0db340a183857d0a787b348681d3 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:53:47 2022 +0100 Don't crash CLI on exceptions commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:11:55 2022 +0100 add missing position_embeddings commit 12769b3d3562ef71e0f54946b532ad077e10043c Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 13:33:25 2022 +0100 debugging why it don't work commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 13:21:33 2022 +0100 debugging why it don't work commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1 Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 12:48:38 2022 +0100 use TextualInversionManager in place of embeddings (wip, doesn't work) commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 12:48:38 2022 +0100 use TextualInversionManager in place of embeddings (wip, doesn't work) commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf Merge: 6e4dad60 023df37e Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:37:31 2022 +0100 Merge branch 'feature_textual_inversion_mgr' into dev/diffusers commit 023df37efffa67434f77def7fc3c9dfb29f699fd Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:36:54 2022 +0100 cleanup commit 05fac594eaf79d0058e3c48deee93df603f136c2 Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:07:49 2022 +0100 tweak error checking commit 009f32ed39a7280997c3ffab112adadee0b44279 Author: damian <null@damianstewart.com> Date: Thu Dec 15 21:29:47 2022 +0100 unit tests passing for embeddings with vector length >1 commit beb1b08d9a98112ed2fe073580568e1a18698da3 Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 13:39:09 2022 +0100 more explicit equality tests when overwriting commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9 Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 13:30:13 2022 +0100 wip textual inversion manager (unit tests passing for 1v embedding overwriting) commit 417c2b57d90924a839616bfb66804faab8039e4c Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 12:30:55 2022 +0100 wip textual inversion manager (unit tests passing for base stuff + padding) commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 10:57:57 2022 +0100 wip new TextualInversionManager * stop using WeightedFrozenCLIPEmbedder * store diffusion models locally - configure_invokeai.py reconfigured to store diffusion models rather than CompVis models - hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id - models.yaml does **NOT** use path, just repo_id - "repo_name" changed to "repo_id" to following hugging face conventions - Models are loaded with full precision pending further work. * allow non-local files during development * path takes priority over repo_id * MVP for model_cache and configure_invokeai - Feature complete (almost) - configure_invokeai.py downloads both .ckpt and diffuser models, along with their VAEs. Both types of download are controlled by a unified INITIAL_MODELS.yaml file. - model_cache can load both type of model and switches back and forth in CPU. No memory leaks detected TO DO: 1. I have not yet turned on the LocalOnly flag for diffuser models, so the code will check the Hugging Face repo for updates before using the locally cached models. This will break firewalled systems. I am thinking of putting in a global check for internet connectivity at startup time and setting the LocalOnly flag based on this. It would be good to check updates if there is connectivity. 2. I have not gone completely through INITIAL_MODELS.yaml to check which models are available as diffusers and which are not. So models like PaperCut and VoxelArt may not load properly. The runway and stability models are checked, as well as the Trinart models. 3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE: 1. When loading a .ckpt file there are lots of messages like this: Warning! ldm.modules.attention.CrossAttention is no longer being maintained. Please use InvokeAICrossAttention instead. I'm not sure how to address this. 2. The ckpt models ***don't actually run*** due to the lack of special-case support for them in the generator objects. For example, here's the hard crash you get when you run txt2img against the legacy waifu-diffusion-1.3 model: ``` >> An error occurred: Traceback (most recent call last): File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main main_loop(gen, opt) File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop gen.prompt2image( File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image results = generator.generate( File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate image = make_image(x_T) File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image pipeline_output = pipeline.image_from_embeddings( File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__ raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings' ``` 3. The inpainting diffusion model isn't working. Here's the output of "banana sushi" when inpainting-1.5 is loaded: ``` Traceback (most recent call last): File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image results = generator.generate( File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate image = make_image(x_T) File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image pipeline_output = pipeline.image_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings result_latents, result_attention_map_saver = self.latents_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings result: PipelineIntermediateState = infer_latents_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__ for result in self.generator_method(*args, **kwargs): File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings step_output = self.step(batched_t, latents, guidance_scale, File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs) File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step pred_original_sample = sample - sigma * model_output RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1 ``` * proper support for float32/float16 - configure script now correctly detects user's preference for fp16/32 and downloads the correct diffuser version. If fp16 version not available, falls back to fp32 version. - misc code cleanup and simplification in model_cache * add on-the-fly conversion of .ckpt to diffusers models 1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py. 2. A new !optimize command has been added to the CLI. Should be ported to Web GUI. User experience on the CLI is this: ``` invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model. This operation will take 30-60s to complete. Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 Writing new config file entry for sd-v1-4... >> New configuration: sd-v1-4: description: Optimized version of sd-v1-4 format: diffusers path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 OK to import [n]? y >> Verifying that new model loads... >> Current VRAM usage: 2.60G >> Offloading stable-diffusion-2.1 to CPU >> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 | Using faster float16 precision You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \ license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\ disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 . | training width x height = (512 x 512) >> Model loaded in 3.48s >> Max VRAM used to load the model: 2.17G >> Current VRAM usage:2.17G >> Textual inversions available: >> Setting Sampler to k_lms (LMSDiscreteScheduler) Keep model loaded? [y] ``` * add parallel set of generator files for ckpt legacy generation * generation using legacy ckpt models now working * diffusers: fix missing attention_maps_callback fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36 * associate legacy CrossAttention with .ckpt models * enable autoconvert New --autoconvert CLI option will scan a designated directory for new .ckpt files, convert them into diffuser models, and import them into models.yaml. Works like this: invoke.py --autoconvert /path/to/weights/directory In ModelCache added two new methods: autoconvert_weights(config_path, weights_directory_path, models_directory_path) convert_and_import(ckpt_path, diffuser_path) * diffusers: update to diffusers 0.11 (from 0.10.2) * fix vae loading & width/height calculation * refactor: encapsulate these conditioning data into one container * diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function * add support for safetensors and accelerate * set local_files_only when internet unreachable * diffusers: fix error-handling path when model repo has no fp16 branch * fix generatorinpaint error Fixes : "ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint' https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318 * quench diffuser safety-checker warning * diffusers: support stochastic DDIM eta parameter * fix conda env creation on macos * fix cross-attention with diffusers 0.11 * diffusers: the VAE needs to be tiling as well as the U-Net * diffusers: comment on subfolders * diffusers: embiggen! * diffusers: make model_cache.list_models serializable * diffusers(inpaint): restore scaling functionality * fix requirements clash between numba and numpy 1.24 * diffusers: allow inpainting model to do non-inpainting tasks * start expanding model_cache functionality * add import_ckpt_model() and import_diffuser_model() methods to model_manager - in addition, model_cache.py is now renamed to model_manager.py * allow "recommended" flag to be optional in INITIAL_MODELS.yaml * configure_invokeai now downloads VAE diffusers in advance * rename ModelCache to ModelManager * remove support for `repo_name` in models.yaml * check for and refuse to load embeddings trained on incompatible models * models.yaml.example: s/repo_name/repo_id and remove extra INITIAL_MODELS now that the main one has diffusers models in it. * add MVP textual inversion script * refactor(InvokeAIDiffuserComponent): factor out _combine() * InvokeAIDiffuserComponent: implement threshold * InvokeAIDiffuserComponent: diagnostic logs for threshold ...this does not look right * add a curses-based frontend to textual inversion - not quite working yet - requires npyscreen installed - on windows will also have the windows-curses requirement, but not added to requirements yet * add curses-based interface for textual inversion * fix crash in convert_and_import() - This corrects a "local variable referenced before assignment" error in model_manager.convert_and_import() * potential workaround for no 'state_dict' key error - As reported in https://github.com/huggingface/diffusers/issues/1876 * create TI output dir if needed * Update environment-lin-cuda.yml (#2159) Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~ * diffusers: update sampler-to-scheduler mapping based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672 * improve user exp for ckt to diffusers conversion - !optimize_models command now operates on an existing ckpt file entry in models.yaml - replaces existing entry, rather than adding a new one - offers to delete the ckpt file after conversion * web: adapt progress callback to deal with old generator or new diffusers pipeline * clean-up model_manager code - add_model() verified to work for .ckpt local paths, .ckpt remote URLs, diffusers local paths, and diffusers repo_ids - convert_and_import() verified to work for local and remove .ckpt files * handle edge cases for import_model() and convert_model() * add support for safetensor .ckpt files * fix name error * code cleanup with pyflake * improve model setting behavior - If the user enters an invalid model name at startup time, will not try to load it, warn, and use default model - CLI UI enhancement: include currently active model in the command line prompt. * update test-invoke-pip.yml - fix model cache path to point to runwayml/stable-diffusion-v1-5 - remove `skip-sd-weights` from configure_invokeai.py args * exclude dev/diffusers from "fail for draft PRs" * disable "fail on PR jobs" * re-add `--skip-sd-weights` since no space * update workflow environments - include `INVOKE_MODEL_RECONFIGURE: '--yes'` * clean up model load failure handling - Allow CLI to run even when no model is defined or loadable. - Inhibit stack trace when model load fails - only show last error - Give user *option* to run configure_invokeai.py when no models successfully load. - Restart invokeai after reconfiguration. * further edge-case handling 1) only one model in models.yaml file, and that model is broken 2) no models in models.yaml 3) models.yaml doesn't exist at all * fix incorrect model status listing - "cached" was not being returned from list_models() - normalize handling of exceptions during model loading: - Passing an invalid model name to generate.set_model() will return a KeyError - All other exceptions are returned as the appropriate Exception * CI: do download weights (if not already cached) * diffusers: fix scheduler loading in offline mode * CI: fix model name (no longer has `diffusers-` prefix) * Update txt2img2img.py (#2256) * fixes to share models with HuggingFace cache system - If HF_HOME environment variable is defined, then all huggingface models are stored in that directory following the standard conventions. - For seamless interoperability, set HF_HOME to ~/.cache/huggingface - If HF_HOME not defined, then models are stored in ~/invokeai/models. This is equivalent to setting HF_HOME to ~/invokeai/models A future commit will add a migration mechanism so that this change doesn't break previous installs. * feat - make model storage compatible with hugging face caching system This commit alters the InvokeAI model directory to be compatible with hugging face, making it easier to share diffusers (and other models) across different programs. - If the HF_HOME environment variable is not set, then models are cached in ~/invokeai/models in a format that is identical to the HuggingFace cache. - If HF_HOME is set, then models are cached wherever HF_HOME points. - To enable sharing with other HuggingFace library clients, set HF_HOME to ~/.cache/huggingface to set the default cache location or to ~/invokeai/models to have huggingface cache inside InvokeAI. * fixes to share models with HuggingFace cache system - If HF_HOME environment variable is defined, then all huggingface models are stored in that directory following the standard conventions. - For seamless interoperability, set HF_HOME to ~/.cache/huggingface - If HF_HOME not defined, then models are stored in ~/invokeai/models. This is equivalent to setting HF_HOME to ~/invokeai/models A future commit will add a migration mechanism so that this change doesn't break previous installs. * fix error "no attribute CkptInpaint" * model_manager.list_models() returns entire model config stanza+status * Initial Draft - Model Manager Diffusers * added hash function to diffusers * implement sha256 hashes on diffusers models * Add Model Manager Support for Diffusers * fix various problems with model manager - in cli import functions, fix not enough values to unpack from _get_name_and_desc() - fix crash when using old-style vae: value with new-style diffuser * rebuild frontend * fix dictconfig-not-serializable issue * fix NoneType' object is not subscriptable crash in model_manager * fix "str has no attribute get" error in model_manager list_models() * Add path and repo_id support for Diffusers Model Manager Also fixes bugs * Fix tooltip IT localization not working * Add Version Number To WebUI * Optimize Model Search * Fix incorrect font on the Model Manager UI * Fix image degradation on merge fixes - [Experimental] This change should effectively fix a couple of things. - Fix image degradation on subsequent merges of the canvas layers. - Fix the slight transparent border that is left behind when filling the bounding box with a color. - Fix the left over line of color when filling a bounding box with color. So far there are no side effects for this. If any, please report. * Add local model filtering for Diffusers / Checkpoints * Go to home on modal close for the Add Modal UI * Styling Fixes * Model Manager Diffusers Localization Update * Add Safe Tensor scanning to Model Manager * Fix model edit form dispatching string values instead of numbers. * Resolve VAE handling / edge cases for supplied repos * defer injecting tokens for textual inversions until they're used for the first time * squash a console warning * implement model migration check * add_model() overwrites previous config rather than merges * fix model config file attribute merging * fix precision handling in textual inversion script * allow ckpt conversion script to work with safetensors .ckpts Applied patch here: https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47 * fix name "args" is not defined crash in textual_inversion_training * fix a second NameError: name 'args' is not defined crash * fix loading of the safety checker from the global cache dir * add installation step to textual inversion frontend - After a successful training run, the script will copy learned_embeds.bin to a subfolder of the embeddings directory. - User given the option to delete the logs and intermediate checkpoints (which together use 7-8G of space) - If textual inversion training fails, reports the error gracefully. * don't crash out on incompatible embeddings - put try: blocks around places where the system tries to load an embedding which is incompatible with the currently loaded model * add support for checkpoint resuming * textual inversion preferences are saved and restored between sessions - Preferences are stored in a file named text-inversion-training/preferences.conf - Currently the resume-from-checkpoint option is not working correctly. Possible bug in textual_inversion_training.py? * copy learned_embeddings.bin into right location * add front end for diffusers model merging - Front end doesn't do anything yet!!!! - Made change to model name parsing in CLI to support ability to have merged models with the "+" character in their names. * improve inpainting experience - recommend ckpt version of inpainting-1.5 to user - fix get_noise() bug in ckpt version of omnibus.py * update environment*yml * tweak instructions to install HuggingFace token * bump version number * enhance update scripts - update scripts will now fetch new INITIAL_MODELS.yaml so that configure_invokeai.py will know about the diffusers versions. * enhance invoke.sh/invoke.bat launchers - added configure_invokeai.py to menu - menu defaults to browser-based invoke * remove conda workflow (#2321) * fix `token_ids has shape torch.Size([79]) - expected [77]` * update CHANGELOG.md with 2.3.* info - Add information on how formats have changed and the upgrade process. - Add short bug list. Co-authored-by: Damian Stewart <d@damianstewart.com> Co-authored-by: Damian Stewart <null@damianstewart.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com> Co-authored-by: mauwii <Mauwii@outlook.de> Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com> Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com> Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com> Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
params:
parameterization: "v"
linear_start: 0.00085
linear_end: 0.0120
num_timesteps_cond: 1
log_every_t: 200
timesteps: 1000
first_stage_key: "jpg"
cond_stage_key: "txt"
image_size: 64
channels: 4
cond_stage_trainable: false
conditioning_key: crossattn
monitor: val/loss_simple_ema
scale_factor: 0.18215
use_ema: False # we set this to false because this is an inference only config
unet_config:
all vestiges of ldm.invoke removed
2023-03-03 06:02:00 +00:00
target: invokeai.backend.stable_diffusion.diffusionmodules.openaimodel.UNetModel
use 🧨diffusers model (#1583) * initial commit of DiffusionPipeline class * spike: proof of concept using diffusers for txt2img * doc: type hints for Generator * refactor(model_cache): factor out load_ckpt * model_cache: add ability to load a diffusers model pipeline and update associated things in Generate & Generator to not instantly fail when that happens * model_cache: fix model default image dimensions * txt2img: support switching diffusers schedulers * diffusers: let the scheduler do its scaling of the initial latents Remove IPNDM scheduler; it is not behaving. * web server: update image_progress callback for diffusers data * diffusers: restore prompt weighting feature * diffusers: fix set-sampler error following model switch * diffusers: use InvokeAIDiffuserComponent for conditioning * cross_attention_control: stub (no-op) implementations for diffusers * model_cache: let offload_model work with DiffusionPipeline, sorta. * models.yaml.example: add diffusers-format model, set as default * test-invoke-conda: use diffusers-format model test-invoke-conda: put huggingface-token where the library can use it * environment-mac: upgrade to diffusers 0.7 (from 0.6) this was already done for linux; mac must have been lost in the merge. * preload_models: explicitly load diffusers models In non-interactive mode too, as long as you're logged in. * fix(model_cache): don't check `model.config` in diffusers format clean-up from recent merge. * diffusers integration: support img2img * dev: upgrade to diffusers 0.8 (from 0.7.1) We get to remove some code by using methods that were factored out in the base class. * refactor: remove backported img2img.get_timesteps now that we can use it directly from diffusers 0.8.1 * ci: use diffusers model * dev: upgrade to diffusers 0.9 (from 0.8.1) * lint: correct annotations for Python 3.9. * lint: correct AttributeError.name reference for Python 3.9. * CI: prefer diffusers-1.4 because it no longer requires a token The RunwayML models still do. * build: there's yet another place to update requirements? * configure: try to download models even without token Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.) * configure: add troubleshooting info for config-not-found * fix(configure): prepend root to config path * fix(configure): remove second `default: true` from models example * CI: simplify test-on-push logic now that we don't need secrets The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks. * create an embedding_manager for diffusers * internal: avoid importing diffusers DummyObject see https://github.com/huggingface/diffusers/issues/1479 * fix "config attributes…not expected" diffusers warnings. * fix deprecated scheduler construction * work around an apparent MPS torch bug that causes conditioning to have no effect * 🚧 post-rebase repair * preliminary support for outpainting (no masking yet) * monkey-patch diffusers.attention and use Invoke lowvram code * add always_use_cpu arg to bypass MPS * add cross-attention control support to diffusers (fails on MPS) For unknown reasons MPS produces garbage output with .swap(). Use --always_use_cpu arg to invoke.py for now to test this code on MPS. * diffusers support for the inpainting model * fix debug_image to not crash with non-RGB images. * inpainting for the normal model [WIP] This seems to be performing well until the LAST STEP, at which point it dissolves to confetti. * fix off-by-one bug in cross-attention-control (#1774) prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness). based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly. * refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary * inpainting for the normal model. I think it works this time. * diffusers: reset num_vectors_per_token sync with 44a00555718f1df173c60da0ed646cf700e29537 * diffusers: txt2img2img (hires_fix) with so much slicing and dicing of pipeline methods to stitch them together * refactor(diffusers): reduce some code duplication amongst the different tasks * fixup! refactor(diffusers): reduce some code duplication amongst the different tasks * diffusers: enable DPMSolver++ scheduler * diffusers: upgrade to diffusers 0.10, add Heun scheduler * diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers * CI: default to diffusers-1.5 now that runwayml token requirement is gone * diffusers: update to 0.10 (and transformers to 4.25) * diffusers: use xformers when available diffusers no longer auto-enables this as of 0.10.2. * diffusers: make masked img2img behave better with multi-step schedulers re-randomizing the noise each step was confusing them. * diffusers: work more better with more models. fixed relative path problem with local models. fixed models on hub not always having a `fp16` branch. * diffusers: stopgap fix for attention_maps_callback crash after recent merge * fixup import merge conflicts correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b * test: add tests/inpainting inputs for masked img2img * diffusers(AddsMaskedGuidance): partial fix for k-schedulers Prevents them from crashing, but results are still hot garbage. * fix --safety_checker arg parsing and add note to diffusers loader about where safety checker gets called * generate: fix import error * CI: don't try to read the old init location * diffusers: support loading an alternate VAE * CI: remove sh-syntax if-statement so it doesn't crash powershell * CI: fold strings in yaml because backslash is not line-continuation in powershell * attention maps callback stuff for diffusers * build: fix syntax error in environment-mac * diffusers: add INITIAL_MODELS with diffusers-compatible repos * re-enable the embedding manager; closes #1778 * Squashed commit of the following: commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:43:07 2022 +0100 import new load handling from EmbeddingManager and cleanup commit c4abe91a5ba0d415b45bf734068385668b7a66e6 Merge: 032e856e 1efc6397 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:09:53 2022 +0100 Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager commit 032e856eefb3bbc39534f5daafd25764bcfcef8b Merge: 8b4f0fe9 bc515e24 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:08:01 2022 +0100 Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager commit 1efc6397fc6e61c1aff4b0258b93089d61de5955 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:04:28 2022 +0100 cleanup and add performance notes commit e400f804ac471a0ca2ba432fd658778b20c7bdab Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 14:45:07 2022 +0100 fix bug and update unit tests commit deb9ae0ae1016750e93ce8275734061f7285a231 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 14:28:29 2022 +0100 textual inversion manager seems to work commit 162e02505dec777e91a983c4d0fb52e950d25ff0 Merge: cbad4583 12769b3d Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:58:03 2022 +0100 Merge branch 'main' into feature_textual_inversion_mgr commit cbad45836c6aace6871a90f2621a953f49433131 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:54:10 2022 +0100 use position embeddings commit 070344c69b0e0db340a183857d0a787b348681d3 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:53:47 2022 +0100 Don't crash CLI on exceptions commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:11:55 2022 +0100 add missing position_embeddings commit 12769b3d3562ef71e0f54946b532ad077e10043c Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 13:33:25 2022 +0100 debugging why it don't work commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 13:21:33 2022 +0100 debugging why it don't work commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1 Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 12:48:38 2022 +0100 use TextualInversionManager in place of embeddings (wip, doesn't work) commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 12:48:38 2022 +0100 use TextualInversionManager in place of embeddings (wip, doesn't work) commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf Merge: 6e4dad60 023df37e Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:37:31 2022 +0100 Merge branch 'feature_textual_inversion_mgr' into dev/diffusers commit 023df37efffa67434f77def7fc3c9dfb29f699fd Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:36:54 2022 +0100 cleanup commit 05fac594eaf79d0058e3c48deee93df603f136c2 Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:07:49 2022 +0100 tweak error checking commit 009f32ed39a7280997c3ffab112adadee0b44279 Author: damian <null@damianstewart.com> Date: Thu Dec 15 21:29:47 2022 +0100 unit tests passing for embeddings with vector length >1 commit beb1b08d9a98112ed2fe073580568e1a18698da3 Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 13:39:09 2022 +0100 more explicit equality tests when overwriting commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9 Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 13:30:13 2022 +0100 wip textual inversion manager (unit tests passing for 1v embedding overwriting) commit 417c2b57d90924a839616bfb66804faab8039e4c Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 12:30:55 2022 +0100 wip textual inversion manager (unit tests passing for base stuff + padding) commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 10:57:57 2022 +0100 wip new TextualInversionManager * stop using WeightedFrozenCLIPEmbedder * store diffusion models locally - configure_invokeai.py reconfigured to store diffusion models rather than CompVis models - hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id - models.yaml does **NOT** use path, just repo_id - "repo_name" changed to "repo_id" to following hugging face conventions - Models are loaded with full precision pending further work. * allow non-local files during development * path takes priority over repo_id * MVP for model_cache and configure_invokeai - Feature complete (almost) - configure_invokeai.py downloads both .ckpt and diffuser models, along with their VAEs. Both types of download are controlled by a unified INITIAL_MODELS.yaml file. - model_cache can load both type of model and switches back and forth in CPU. No memory leaks detected TO DO: 1. I have not yet turned on the LocalOnly flag for diffuser models, so the code will check the Hugging Face repo for updates before using the locally cached models. This will break firewalled systems. I am thinking of putting in a global check for internet connectivity at startup time and setting the LocalOnly flag based on this. It would be good to check updates if there is connectivity. 2. I have not gone completely through INITIAL_MODELS.yaml to check which models are available as diffusers and which are not. So models like PaperCut and VoxelArt may not load properly. The runway and stability models are checked, as well as the Trinart models. 3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE: 1. When loading a .ckpt file there are lots of messages like this: Warning! ldm.modules.attention.CrossAttention is no longer being maintained. Please use InvokeAICrossAttention instead. I'm not sure how to address this. 2. The ckpt models ***don't actually run*** due to the lack of special-case support for them in the generator objects. For example, here's the hard crash you get when you run txt2img against the legacy waifu-diffusion-1.3 model: ``` >> An error occurred: Traceback (most recent call last): File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main main_loop(gen, opt) File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop gen.prompt2image( File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image results = generator.generate( File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate image = make_image(x_T) File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image pipeline_output = pipeline.image_from_embeddings( File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__ raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings' ``` 3. The inpainting diffusion model isn't working. Here's the output of "banana sushi" when inpainting-1.5 is loaded: ``` Traceback (most recent call last): File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image results = generator.generate( File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate image = make_image(x_T) File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image pipeline_output = pipeline.image_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings result_latents, result_attention_map_saver = self.latents_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings result: PipelineIntermediateState = infer_latents_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__ for result in self.generator_method(*args, **kwargs): File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings step_output = self.step(batched_t, latents, guidance_scale, File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs) File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step pred_original_sample = sample - sigma * model_output RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1 ``` * proper support for float32/float16 - configure script now correctly detects user's preference for fp16/32 and downloads the correct diffuser version. If fp16 version not available, falls back to fp32 version. - misc code cleanup and simplification in model_cache * add on-the-fly conversion of .ckpt to diffusers models 1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py. 2. A new !optimize command has been added to the CLI. Should be ported to Web GUI. User experience on the CLI is this: ``` invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model. This operation will take 30-60s to complete. Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 Writing new config file entry for sd-v1-4... >> New configuration: sd-v1-4: description: Optimized version of sd-v1-4 format: diffusers path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 OK to import [n]? y >> Verifying that new model loads... >> Current VRAM usage: 2.60G >> Offloading stable-diffusion-2.1 to CPU >> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 | Using faster float16 precision You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \ license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\ disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 . | training width x height = (512 x 512) >> Model loaded in 3.48s >> Max VRAM used to load the model: 2.17G >> Current VRAM usage:2.17G >> Textual inversions available: >> Setting Sampler to k_lms (LMSDiscreteScheduler) Keep model loaded? [y] ``` * add parallel set of generator files for ckpt legacy generation * generation using legacy ckpt models now working * diffusers: fix missing attention_maps_callback fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36 * associate legacy CrossAttention with .ckpt models * enable autoconvert New --autoconvert CLI option will scan a designated directory for new .ckpt files, convert them into diffuser models, and import them into models.yaml. Works like this: invoke.py --autoconvert /path/to/weights/directory In ModelCache added two new methods: autoconvert_weights(config_path, weights_directory_path, models_directory_path) convert_and_import(ckpt_path, diffuser_path) * diffusers: update to diffusers 0.11 (from 0.10.2) * fix vae loading & width/height calculation * refactor: encapsulate these conditioning data into one container * diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function * add support for safetensors and accelerate * set local_files_only when internet unreachable * diffusers: fix error-handling path when model repo has no fp16 branch * fix generatorinpaint error Fixes : "ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint' https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318 * quench diffuser safety-checker warning * diffusers: support stochastic DDIM eta parameter * fix conda env creation on macos * fix cross-attention with diffusers 0.11 * diffusers: the VAE needs to be tiling as well as the U-Net * diffusers: comment on subfolders * diffusers: embiggen! * diffusers: make model_cache.list_models serializable * diffusers(inpaint): restore scaling functionality * fix requirements clash between numba and numpy 1.24 * diffusers: allow inpainting model to do non-inpainting tasks * start expanding model_cache functionality * add import_ckpt_model() and import_diffuser_model() methods to model_manager - in addition, model_cache.py is now renamed to model_manager.py * allow "recommended" flag to be optional in INITIAL_MODELS.yaml * configure_invokeai now downloads VAE diffusers in advance * rename ModelCache to ModelManager * remove support for `repo_name` in models.yaml * check for and refuse to load embeddings trained on incompatible models * models.yaml.example: s/repo_name/repo_id and remove extra INITIAL_MODELS now that the main one has diffusers models in it. * add MVP textual inversion script * refactor(InvokeAIDiffuserComponent): factor out _combine() * InvokeAIDiffuserComponent: implement threshold * InvokeAIDiffuserComponent: diagnostic logs for threshold ...this does not look right * add a curses-based frontend to textual inversion - not quite working yet - requires npyscreen installed - on windows will also have the windows-curses requirement, but not added to requirements yet * add curses-based interface for textual inversion * fix crash in convert_and_import() - This corrects a "local variable referenced before assignment" error in model_manager.convert_and_import() * potential workaround for no 'state_dict' key error - As reported in https://github.com/huggingface/diffusers/issues/1876 * create TI output dir if needed * Update environment-lin-cuda.yml (#2159) Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~ * diffusers: update sampler-to-scheduler mapping based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672 * improve user exp for ckt to diffusers conversion - !optimize_models command now operates on an existing ckpt file entry in models.yaml - replaces existing entry, rather than adding a new one - offers to delete the ckpt file after conversion * web: adapt progress callback to deal with old generator or new diffusers pipeline * clean-up model_manager code - add_model() verified to work for .ckpt local paths, .ckpt remote URLs, diffusers local paths, and diffusers repo_ids - convert_and_import() verified to work for local and remove .ckpt files * handle edge cases for import_model() and convert_model() * add support for safetensor .ckpt files * fix name error * code cleanup with pyflake * improve model setting behavior - If the user enters an invalid model name at startup time, will not try to load it, warn, and use default model - CLI UI enhancement: include currently active model in the command line prompt. * update test-invoke-pip.yml - fix model cache path to point to runwayml/stable-diffusion-v1-5 - remove `skip-sd-weights` from configure_invokeai.py args * exclude dev/diffusers from "fail for draft PRs" * disable "fail on PR jobs" * re-add `--skip-sd-weights` since no space * update workflow environments - include `INVOKE_MODEL_RECONFIGURE: '--yes'` * clean up model load failure handling - Allow CLI to run even when no model is defined or loadable. - Inhibit stack trace when model load fails - only show last error - Give user *option* to run configure_invokeai.py when no models successfully load. - Restart invokeai after reconfiguration. * further edge-case handling 1) only one model in models.yaml file, and that model is broken 2) no models in models.yaml 3) models.yaml doesn't exist at all * fix incorrect model status listing - "cached" was not being returned from list_models() - normalize handling of exceptions during model loading: - Passing an invalid model name to generate.set_model() will return a KeyError - All other exceptions are returned as the appropriate Exception * CI: do download weights (if not already cached) * diffusers: fix scheduler loading in offline mode * CI: fix model name (no longer has `diffusers-` prefix) * Update txt2img2img.py (#2256) * fixes to share models with HuggingFace cache system - If HF_HOME environment variable is defined, then all huggingface models are stored in that directory following the standard conventions. - For seamless interoperability, set HF_HOME to ~/.cache/huggingface - If HF_HOME not defined, then models are stored in ~/invokeai/models. This is equivalent to setting HF_HOME to ~/invokeai/models A future commit will add a migration mechanism so that this change doesn't break previous installs. * feat - make model storage compatible with hugging face caching system This commit alters the InvokeAI model directory to be compatible with hugging face, making it easier to share diffusers (and other models) across different programs. - If the HF_HOME environment variable is not set, then models are cached in ~/invokeai/models in a format that is identical to the HuggingFace cache. - If HF_HOME is set, then models are cached wherever HF_HOME points. - To enable sharing with other HuggingFace library clients, set HF_HOME to ~/.cache/huggingface to set the default cache location or to ~/invokeai/models to have huggingface cache inside InvokeAI. * fixes to share models with HuggingFace cache system - If HF_HOME environment variable is defined, then all huggingface models are stored in that directory following the standard conventions. - For seamless interoperability, set HF_HOME to ~/.cache/huggingface - If HF_HOME not defined, then models are stored in ~/invokeai/models. This is equivalent to setting HF_HOME to ~/invokeai/models A future commit will add a migration mechanism so that this change doesn't break previous installs. * fix error "no attribute CkptInpaint" * model_manager.list_models() returns entire model config stanza+status * Initial Draft - Model Manager Diffusers * added hash function to diffusers * implement sha256 hashes on diffusers models * Add Model Manager Support for Diffusers * fix various problems with model manager - in cli import functions, fix not enough values to unpack from _get_name_and_desc() - fix crash when using old-style vae: value with new-style diffuser * rebuild frontend * fix dictconfig-not-serializable issue * fix NoneType' object is not subscriptable crash in model_manager * fix "str has no attribute get" error in model_manager list_models() * Add path and repo_id support for Diffusers Model Manager Also fixes bugs * Fix tooltip IT localization not working * Add Version Number To WebUI * Optimize Model Search * Fix incorrect font on the Model Manager UI * Fix image degradation on merge fixes - [Experimental] This change should effectively fix a couple of things. - Fix image degradation on subsequent merges of the canvas layers. - Fix the slight transparent border that is left behind when filling the bounding box with a color. - Fix the left over line of color when filling a bounding box with color. So far there are no side effects for this. If any, please report. * Add local model filtering for Diffusers / Checkpoints * Go to home on modal close for the Add Modal UI * Styling Fixes * Model Manager Diffusers Localization Update * Add Safe Tensor scanning to Model Manager * Fix model edit form dispatching string values instead of numbers. * Resolve VAE handling / edge cases for supplied repos * defer injecting tokens for textual inversions until they're used for the first time * squash a console warning * implement model migration check * add_model() overwrites previous config rather than merges * fix model config file attribute merging * fix precision handling in textual inversion script * allow ckpt conversion script to work with safetensors .ckpts Applied patch here: https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47 * fix name "args" is not defined crash in textual_inversion_training * fix a second NameError: name 'args' is not defined crash * fix loading of the safety checker from the global cache dir * add installation step to textual inversion frontend - After a successful training run, the script will copy learned_embeds.bin to a subfolder of the embeddings directory. - User given the option to delete the logs and intermediate checkpoints (which together use 7-8G of space) - If textual inversion training fails, reports the error gracefully. * don't crash out on incompatible embeddings - put try: blocks around places where the system tries to load an embedding which is incompatible with the currently loaded model * add support for checkpoint resuming * textual inversion preferences are saved and restored between sessions - Preferences are stored in a file named text-inversion-training/preferences.conf - Currently the resume-from-checkpoint option is not working correctly. Possible bug in textual_inversion_training.py? * copy learned_embeddings.bin into right location * add front end for diffusers model merging - Front end doesn't do anything yet!!!! - Made change to model name parsing in CLI to support ability to have merged models with the "+" character in their names. * improve inpainting experience - recommend ckpt version of inpainting-1.5 to user - fix get_noise() bug in ckpt version of omnibus.py * update environment*yml * tweak instructions to install HuggingFace token * bump version number * enhance update scripts - update scripts will now fetch new INITIAL_MODELS.yaml so that configure_invokeai.py will know about the diffusers versions. * enhance invoke.sh/invoke.bat launchers - added configure_invokeai.py to menu - menu defaults to browser-based invoke * remove conda workflow (#2321) * fix `token_ids has shape torch.Size([79]) - expected [77]` * update CHANGELOG.md with 2.3.* info - Add information on how formats have changed and the upgrade process. - Add short bug list. Co-authored-by: Damian Stewart <d@damianstewart.com> Co-authored-by: Damian Stewart <null@damianstewart.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com> Co-authored-by: mauwii <Mauwii@outlook.de> Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com> Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com> Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com> Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
params:
use_checkpoint: True
use_fp16: True
image_size: 32 # unused
in_channels: 4
out_channels: 4
model_channels: 320
attention_resolutions: [ 4, 2, 1 ]
num_res_blocks: 2
channel_mult: [ 1, 2, 4, 4 ]
num_head_channels: 64 # need to fix for flash-attn
use_spatial_transformer: True
use_linear_in_transformer: True
transformer_depth: 1
context_dim: 1024
legacy: False
first_stage_config:
all vestiges of ldm.invoke removed
2023-03-03 06:02:00 +00:00
target: invokeai.backend.stable_diffusion.autoencoder.AutoencoderKL
use 🧨diffusers model (#1583) * initial commit of DiffusionPipeline class * spike: proof of concept using diffusers for txt2img * doc: type hints for Generator * refactor(model_cache): factor out load_ckpt * model_cache: add ability to load a diffusers model pipeline and update associated things in Generate & Generator to not instantly fail when that happens * model_cache: fix model default image dimensions * txt2img: support switching diffusers schedulers * diffusers: let the scheduler do its scaling of the initial latents Remove IPNDM scheduler; it is not behaving. * web server: update image_progress callback for diffusers data * diffusers: restore prompt weighting feature * diffusers: fix set-sampler error following model switch * diffusers: use InvokeAIDiffuserComponent for conditioning * cross_attention_control: stub (no-op) implementations for diffusers * model_cache: let offload_model work with DiffusionPipeline, sorta. * models.yaml.example: add diffusers-format model, set as default * test-invoke-conda: use diffusers-format model test-invoke-conda: put huggingface-token where the library can use it * environment-mac: upgrade to diffusers 0.7 (from 0.6) this was already done for linux; mac must have been lost in the merge. * preload_models: explicitly load diffusers models In non-interactive mode too, as long as you're logged in. * fix(model_cache): don't check `model.config` in diffusers format clean-up from recent merge. * diffusers integration: support img2img * dev: upgrade to diffusers 0.8 (from 0.7.1) We get to remove some code by using methods that were factored out in the base class. * refactor: remove backported img2img.get_timesteps now that we can use it directly from diffusers 0.8.1 * ci: use diffusers model * dev: upgrade to diffusers 0.9 (from 0.8.1) * lint: correct annotations for Python 3.9. * lint: correct AttributeError.name reference for Python 3.9. * CI: prefer diffusers-1.4 because it no longer requires a token The RunwayML models still do. * build: there's yet another place to update requirements? * configure: try to download models even without token Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.) * configure: add troubleshooting info for config-not-found * fix(configure): prepend root to config path * fix(configure): remove second `default: true` from models example * CI: simplify test-on-push logic now that we don't need secrets The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks. * create an embedding_manager for diffusers * internal: avoid importing diffusers DummyObject see https://github.com/huggingface/diffusers/issues/1479 * fix "config attributes…not expected" diffusers warnings. * fix deprecated scheduler construction * work around an apparent MPS torch bug that causes conditioning to have no effect * 🚧 post-rebase repair * preliminary support for outpainting (no masking yet) * monkey-patch diffusers.attention and use Invoke lowvram code * add always_use_cpu arg to bypass MPS * add cross-attention control support to diffusers (fails on MPS) For unknown reasons MPS produces garbage output with .swap(). Use --always_use_cpu arg to invoke.py for now to test this code on MPS. * diffusers support for the inpainting model * fix debug_image to not crash with non-RGB images. * inpainting for the normal model [WIP] This seems to be performing well until the LAST STEP, at which point it dissolves to confetti. * fix off-by-one bug in cross-attention-control (#1774) prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness). based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly. * refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary * inpainting for the normal model. I think it works this time. * diffusers: reset num_vectors_per_token sync with 44a00555718f1df173c60da0ed646cf700e29537 * diffusers: txt2img2img (hires_fix) with so much slicing and dicing of pipeline methods to stitch them together * refactor(diffusers): reduce some code duplication amongst the different tasks * fixup! refactor(diffusers): reduce some code duplication amongst the different tasks * diffusers: enable DPMSolver++ scheduler * diffusers: upgrade to diffusers 0.10, add Heun scheduler * diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers * CI: default to diffusers-1.5 now that runwayml token requirement is gone * diffusers: update to 0.10 (and transformers to 4.25) * diffusers: use xformers when available diffusers no longer auto-enables this as of 0.10.2. * diffusers: make masked img2img behave better with multi-step schedulers re-randomizing the noise each step was confusing them. * diffusers: work more better with more models. fixed relative path problem with local models. fixed models on hub not always having a `fp16` branch. * diffusers: stopgap fix for attention_maps_callback crash after recent merge * fixup import merge conflicts correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b * test: add tests/inpainting inputs for masked img2img * diffusers(AddsMaskedGuidance): partial fix for k-schedulers Prevents them from crashing, but results are still hot garbage. * fix --safety_checker arg parsing and add note to diffusers loader about where safety checker gets called * generate: fix import error * CI: don't try to read the old init location * diffusers: support loading an alternate VAE * CI: remove sh-syntax if-statement so it doesn't crash powershell * CI: fold strings in yaml because backslash is not line-continuation in powershell * attention maps callback stuff for diffusers * build: fix syntax error in environment-mac * diffusers: add INITIAL_MODELS with diffusers-compatible repos * re-enable the embedding manager; closes #1778 * Squashed commit of the following: commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:43:07 2022 +0100 import new load handling from EmbeddingManager and cleanup commit c4abe91a5ba0d415b45bf734068385668b7a66e6 Merge: 032e856e 1efc6397 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:09:53 2022 +0100 Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager commit 032e856eefb3bbc39534f5daafd25764bcfcef8b Merge: 8b4f0fe9 bc515e24 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:08:01 2022 +0100 Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager commit 1efc6397fc6e61c1aff4b0258b93089d61de5955 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:04:28 2022 +0100 cleanup and add performance notes commit e400f804ac471a0ca2ba432fd658778b20c7bdab Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 14:45:07 2022 +0100 fix bug and update unit tests commit deb9ae0ae1016750e93ce8275734061f7285a231 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 14:28:29 2022 +0100 textual inversion manager seems to work commit 162e02505dec777e91a983c4d0fb52e950d25ff0 Merge: cbad4583 12769b3d Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:58:03 2022 +0100 Merge branch 'main' into feature_textual_inversion_mgr commit cbad45836c6aace6871a90f2621a953f49433131 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:54:10 2022 +0100 use position embeddings commit 070344c69b0e0db340a183857d0a787b348681d3 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:53:47 2022 +0100 Don't crash CLI on exceptions commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:11:55 2022 +0100 add missing position_embeddings commit 12769b3d3562ef71e0f54946b532ad077e10043c Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 13:33:25 2022 +0100 debugging why it don't work commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 13:21:33 2022 +0100 debugging why it don't work commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1 Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 12:48:38 2022 +0100 use TextualInversionManager in place of embeddings (wip, doesn't work) commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 12:48:38 2022 +0100 use TextualInversionManager in place of embeddings (wip, doesn't work) commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf Merge: 6e4dad60 023df37e Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:37:31 2022 +0100 Merge branch 'feature_textual_inversion_mgr' into dev/diffusers commit 023df37efffa67434f77def7fc3c9dfb29f699fd Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:36:54 2022 +0100 cleanup commit 05fac594eaf79d0058e3c48deee93df603f136c2 Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:07:49 2022 +0100 tweak error checking commit 009f32ed39a7280997c3ffab112adadee0b44279 Author: damian <null@damianstewart.com> Date: Thu Dec 15 21:29:47 2022 +0100 unit tests passing for embeddings with vector length >1 commit beb1b08d9a98112ed2fe073580568e1a18698da3 Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 13:39:09 2022 +0100 more explicit equality tests when overwriting commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9 Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 13:30:13 2022 +0100 wip textual inversion manager (unit tests passing for 1v embedding overwriting) commit 417c2b57d90924a839616bfb66804faab8039e4c Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 12:30:55 2022 +0100 wip textual inversion manager (unit tests passing for base stuff + padding) commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 10:57:57 2022 +0100 wip new TextualInversionManager * stop using WeightedFrozenCLIPEmbedder * store diffusion models locally - configure_invokeai.py reconfigured to store diffusion models rather than CompVis models - hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id - models.yaml does **NOT** use path, just repo_id - "repo_name" changed to "repo_id" to following hugging face conventions - Models are loaded with full precision pending further work. * allow non-local files during development * path takes priority over repo_id * MVP for model_cache and configure_invokeai - Feature complete (almost) - configure_invokeai.py downloads both .ckpt and diffuser models, along with their VAEs. Both types of download are controlled by a unified INITIAL_MODELS.yaml file. - model_cache can load both type of model and switches back and forth in CPU. No memory leaks detected TO DO: 1. I have not yet turned on the LocalOnly flag for diffuser models, so the code will check the Hugging Face repo for updates before using the locally cached models. This will break firewalled systems. I am thinking of putting in a global check for internet connectivity at startup time and setting the LocalOnly flag based on this. It would be good to check updates if there is connectivity. 2. I have not gone completely through INITIAL_MODELS.yaml to check which models are available as diffusers and which are not. So models like PaperCut and VoxelArt may not load properly. The runway and stability models are checked, as well as the Trinart models. 3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE: 1. When loading a .ckpt file there are lots of messages like this: Warning! ldm.modules.attention.CrossAttention is no longer being maintained. Please use InvokeAICrossAttention instead. I'm not sure how to address this. 2. The ckpt models ***don't actually run*** due to the lack of special-case support for them in the generator objects. For example, here's the hard crash you get when you run txt2img against the legacy waifu-diffusion-1.3 model: ``` >> An error occurred: Traceback (most recent call last): File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main main_loop(gen, opt) File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop gen.prompt2image( File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image results = generator.generate( File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate image = make_image(x_T) File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image pipeline_output = pipeline.image_from_embeddings( File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__ raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings' ``` 3. The inpainting diffusion model isn't working. Here's the output of "banana sushi" when inpainting-1.5 is loaded: ``` Traceback (most recent call last): File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image results = generator.generate( File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate image = make_image(x_T) File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image pipeline_output = pipeline.image_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings result_latents, result_attention_map_saver = self.latents_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings result: PipelineIntermediateState = infer_latents_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__ for result in self.generator_method(*args, **kwargs): File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings step_output = self.step(batched_t, latents, guidance_scale, File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs) File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step pred_original_sample = sample - sigma * model_output RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1 ``` * proper support for float32/float16 - configure script now correctly detects user's preference for fp16/32 and downloads the correct diffuser version. If fp16 version not available, falls back to fp32 version. - misc code cleanup and simplification in model_cache * add on-the-fly conversion of .ckpt to diffusers models 1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py. 2. A new !optimize command has been added to the CLI. Should be ported to Web GUI. User experience on the CLI is this: ``` invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model. This operation will take 30-60s to complete. Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 Writing new config file entry for sd-v1-4... >> New configuration: sd-v1-4: description: Optimized version of sd-v1-4 format: diffusers path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 OK to import [n]? y >> Verifying that new model loads... >> Current VRAM usage: 2.60G >> Offloading stable-diffusion-2.1 to CPU >> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 | Using faster float16 precision You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \ license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\ disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 . | training width x height = (512 x 512) >> Model loaded in 3.48s >> Max VRAM used to load the model: 2.17G >> Current VRAM usage:2.17G >> Textual inversions available: >> Setting Sampler to k_lms (LMSDiscreteScheduler) Keep model loaded? [y] ``` * add parallel set of generator files for ckpt legacy generation * generation using legacy ckpt models now working * diffusers: fix missing attention_maps_callback fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36 * associate legacy CrossAttention with .ckpt models * enable autoconvert New --autoconvert CLI option will scan a designated directory for new .ckpt files, convert them into diffuser models, and import them into models.yaml. Works like this: invoke.py --autoconvert /path/to/weights/directory In ModelCache added two new methods: autoconvert_weights(config_path, weights_directory_path, models_directory_path) convert_and_import(ckpt_path, diffuser_path) * diffusers: update to diffusers 0.11 (from 0.10.2) * fix vae loading & width/height calculation * refactor: encapsulate these conditioning data into one container * diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function * add support for safetensors and accelerate * set local_files_only when internet unreachable * diffusers: fix error-handling path when model repo has no fp16 branch * fix generatorinpaint error Fixes : "ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint' https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318 * quench diffuser safety-checker warning * diffusers: support stochastic DDIM eta parameter * fix conda env creation on macos * fix cross-attention with diffusers 0.11 * diffusers: the VAE needs to be tiling as well as the U-Net * diffusers: comment on subfolders * diffusers: embiggen! * diffusers: make model_cache.list_models serializable * diffusers(inpaint): restore scaling functionality * fix requirements clash between numba and numpy 1.24 * diffusers: allow inpainting model to do non-inpainting tasks * start expanding model_cache functionality * add import_ckpt_model() and import_diffuser_model() methods to model_manager - in addition, model_cache.py is now renamed to model_manager.py * allow "recommended" flag to be optional in INITIAL_MODELS.yaml * configure_invokeai now downloads VAE diffusers in advance * rename ModelCache to ModelManager * remove support for `repo_name` in models.yaml * check for and refuse to load embeddings trained on incompatible models * models.yaml.example: s/repo_name/repo_id and remove extra INITIAL_MODELS now that the main one has diffusers models in it. * add MVP textual inversion script * refactor(InvokeAIDiffuserComponent): factor out _combine() * InvokeAIDiffuserComponent: implement threshold * InvokeAIDiffuserComponent: diagnostic logs for threshold ...this does not look right * add a curses-based frontend to textual inversion - not quite working yet - requires npyscreen installed - on windows will also have the windows-curses requirement, but not added to requirements yet * add curses-based interface for textual inversion * fix crash in convert_and_import() - This corrects a "local variable referenced before assignment" error in model_manager.convert_and_import() * potential workaround for no 'state_dict' key error - As reported in https://github.com/huggingface/diffusers/issues/1876 * create TI output dir if needed * Update environment-lin-cuda.yml (#2159) Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~ * diffusers: update sampler-to-scheduler mapping based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672 * improve user exp for ckt to diffusers conversion - !optimize_models command now operates on an existing ckpt file entry in models.yaml - replaces existing entry, rather than adding a new one - offers to delete the ckpt file after conversion * web: adapt progress callback to deal with old generator or new diffusers pipeline * clean-up model_manager code - add_model() verified to work for .ckpt local paths, .ckpt remote URLs, diffusers local paths, and diffusers repo_ids - convert_and_import() verified to work for local and remove .ckpt files * handle edge cases for import_model() and convert_model() * add support for safetensor .ckpt files * fix name error * code cleanup with pyflake * improve model setting behavior - If the user enters an invalid model name at startup time, will not try to load it, warn, and use default model - CLI UI enhancement: include currently active model in the command line prompt. * update test-invoke-pip.yml - fix model cache path to point to runwayml/stable-diffusion-v1-5 - remove `skip-sd-weights` from configure_invokeai.py args * exclude dev/diffusers from "fail for draft PRs" * disable "fail on PR jobs" * re-add `--skip-sd-weights` since no space * update workflow environments - include `INVOKE_MODEL_RECONFIGURE: '--yes'` * clean up model load failure handling - Allow CLI to run even when no model is defined or loadable. - Inhibit stack trace when model load fails - only show last error - Give user *option* to run configure_invokeai.py when no models successfully load. - Restart invokeai after reconfiguration. * further edge-case handling 1) only one model in models.yaml file, and that model is broken 2) no models in models.yaml 3) models.yaml doesn't exist at all * fix incorrect model status listing - "cached" was not being returned from list_models() - normalize handling of exceptions during model loading: - Passing an invalid model name to generate.set_model() will return a KeyError - All other exceptions are returned as the appropriate Exception * CI: do download weights (if not already cached) * diffusers: fix scheduler loading in offline mode * CI: fix model name (no longer has `diffusers-` prefix) * Update txt2img2img.py (#2256) * fixes to share models with HuggingFace cache system - If HF_HOME environment variable is defined, then all huggingface models are stored in that directory following the standard conventions. - For seamless interoperability, set HF_HOME to ~/.cache/huggingface - If HF_HOME not defined, then models are stored in ~/invokeai/models. This is equivalent to setting HF_HOME to ~/invokeai/models A future commit will add a migration mechanism so that this change doesn't break previous installs. * feat - make model storage compatible with hugging face caching system This commit alters the InvokeAI model directory to be compatible with hugging face, making it easier to share diffusers (and other models) across different programs. - If the HF_HOME environment variable is not set, then models are cached in ~/invokeai/models in a format that is identical to the HuggingFace cache. - If HF_HOME is set, then models are cached wherever HF_HOME points. - To enable sharing with other HuggingFace library clients, set HF_HOME to ~/.cache/huggingface to set the default cache location or to ~/invokeai/models to have huggingface cache inside InvokeAI. * fixes to share models with HuggingFace cache system - If HF_HOME environment variable is defined, then all huggingface models are stored in that directory following the standard conventions. - For seamless interoperability, set HF_HOME to ~/.cache/huggingface - If HF_HOME not defined, then models are stored in ~/invokeai/models. This is equivalent to setting HF_HOME to ~/invokeai/models A future commit will add a migration mechanism so that this change doesn't break previous installs. * fix error "no attribute CkptInpaint" * model_manager.list_models() returns entire model config stanza+status * Initial Draft - Model Manager Diffusers * added hash function to diffusers * implement sha256 hashes on diffusers models * Add Model Manager Support for Diffusers * fix various problems with model manager - in cli import functions, fix not enough values to unpack from _get_name_and_desc() - fix crash when using old-style vae: value with new-style diffuser * rebuild frontend * fix dictconfig-not-serializable issue * fix NoneType' object is not subscriptable crash in model_manager * fix "str has no attribute get" error in model_manager list_models() * Add path and repo_id support for Diffusers Model Manager Also fixes bugs * Fix tooltip IT localization not working * Add Version Number To WebUI * Optimize Model Search * Fix incorrect font on the Model Manager UI * Fix image degradation on merge fixes - [Experimental] This change should effectively fix a couple of things. - Fix image degradation on subsequent merges of the canvas layers. - Fix the slight transparent border that is left behind when filling the bounding box with a color. - Fix the left over line of color when filling a bounding box with color. So far there are no side effects for this. If any, please report. * Add local model filtering for Diffusers / Checkpoints * Go to home on modal close for the Add Modal UI * Styling Fixes * Model Manager Diffusers Localization Update * Add Safe Tensor scanning to Model Manager * Fix model edit form dispatching string values instead of numbers. * Resolve VAE handling / edge cases for supplied repos * defer injecting tokens for textual inversions until they're used for the first time * squash a console warning * implement model migration check * add_model() overwrites previous config rather than merges * fix model config file attribute merging * fix precision handling in textual inversion script * allow ckpt conversion script to work with safetensors .ckpts Applied patch here: https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47 * fix name "args" is not defined crash in textual_inversion_training * fix a second NameError: name 'args' is not defined crash * fix loading of the safety checker from the global cache dir * add installation step to textual inversion frontend - After a successful training run, the script will copy learned_embeds.bin to a subfolder of the embeddings directory. - User given the option to delete the logs and intermediate checkpoints (which together use 7-8G of space) - If textual inversion training fails, reports the error gracefully. * don't crash out on incompatible embeddings - put try: blocks around places where the system tries to load an embedding which is incompatible with the currently loaded model * add support for checkpoint resuming * textual inversion preferences are saved and restored between sessions - Preferences are stored in a file named text-inversion-training/preferences.conf - Currently the resume-from-checkpoint option is not working correctly. Possible bug in textual_inversion_training.py? * copy learned_embeddings.bin into right location * add front end for diffusers model merging - Front end doesn't do anything yet!!!! - Made change to model name parsing in CLI to support ability to have merged models with the "+" character in their names. * improve inpainting experience - recommend ckpt version of inpainting-1.5 to user - fix get_noise() bug in ckpt version of omnibus.py * update environment*yml * tweak instructions to install HuggingFace token * bump version number * enhance update scripts - update scripts will now fetch new INITIAL_MODELS.yaml so that configure_invokeai.py will know about the diffusers versions. * enhance invoke.sh/invoke.bat launchers - added configure_invokeai.py to menu - menu defaults to browser-based invoke * remove conda workflow (#2321) * fix `token_ids has shape torch.Size([79]) - expected [77]` * update CHANGELOG.md with 2.3.* info - Add information on how formats have changed and the upgrade process. - Add short bug list. Co-authored-by: Damian Stewart <d@damianstewart.com> Co-authored-by: Damian Stewart <null@damianstewart.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com> Co-authored-by: mauwii <Mauwii@outlook.de> Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com> Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com> Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com> Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
params:
embed_dim: 4
monitor: val/rec_loss
ddconfig:
#attn_type: "vanilla-xformers"
double_z: true
z_channels: 4
resolution: 256
in_channels: 3
out_ch: 3
ch: 128
ch_mult:
- 1
- 2
- 4
- 4
num_res_blocks: 2
attn_resolutions: []
dropout: 0.0
lossconfig:
target: torch.nn.Identity
cond_stage_config:
all vestiges of ldm.invoke removed
2023-03-03 06:02:00 +00:00
target: invokeai.backend.stable_diffusion.encoders.modules.FrozenOpenCLIPEmbedder
use 🧨diffusers model (#1583) * initial commit of DiffusionPipeline class * spike: proof of concept using diffusers for txt2img * doc: type hints for Generator * refactor(model_cache): factor out load_ckpt * model_cache: add ability to load a diffusers model pipeline and update associated things in Generate & Generator to not instantly fail when that happens * model_cache: fix model default image dimensions * txt2img: support switching diffusers schedulers * diffusers: let the scheduler do its scaling of the initial latents Remove IPNDM scheduler; it is not behaving. * web server: update image_progress callback for diffusers data * diffusers: restore prompt weighting feature * diffusers: fix set-sampler error following model switch * diffusers: use InvokeAIDiffuserComponent for conditioning * cross_attention_control: stub (no-op) implementations for diffusers * model_cache: let offload_model work with DiffusionPipeline, sorta. * models.yaml.example: add diffusers-format model, set as default * test-invoke-conda: use diffusers-format model test-invoke-conda: put huggingface-token where the library can use it * environment-mac: upgrade to diffusers 0.7 (from 0.6) this was already done for linux; mac must have been lost in the merge. * preload_models: explicitly load diffusers models In non-interactive mode too, as long as you're logged in. * fix(model_cache): don't check `model.config` in diffusers format clean-up from recent merge. * diffusers integration: support img2img * dev: upgrade to diffusers 0.8 (from 0.7.1) We get to remove some code by using methods that were factored out in the base class. * refactor: remove backported img2img.get_timesteps now that we can use it directly from diffusers 0.8.1 * ci: use diffusers model * dev: upgrade to diffusers 0.9 (from 0.8.1) * lint: correct annotations for Python 3.9. * lint: correct AttributeError.name reference for Python 3.9. * CI: prefer diffusers-1.4 because it no longer requires a token The RunwayML models still do. * build: there's yet another place to update requirements? * configure: try to download models even without token Models in the CompVis and stabilityai repos no longer require them. (But runwayml still does.) * configure: add troubleshooting info for config-not-found * fix(configure): prepend root to config path * fix(configure): remove second `default: true` from models example * CI: simplify test-on-push logic now that we don't need secrets The "test on push but only in forks" logic was only necessary when tests didn't work for PRs-from-forks. * create an embedding_manager for diffusers * internal: avoid importing diffusers DummyObject see https://github.com/huggingface/diffusers/issues/1479 * fix "config attributes…not expected" diffusers warnings. * fix deprecated scheduler construction * work around an apparent MPS torch bug that causes conditioning to have no effect * 🚧 post-rebase repair * preliminary support for outpainting (no masking yet) * monkey-patch diffusers.attention and use Invoke lowvram code * add always_use_cpu arg to bypass MPS * add cross-attention control support to diffusers (fails on MPS) For unknown reasons MPS produces garbage output with .swap(). Use --always_use_cpu arg to invoke.py for now to test this code on MPS. * diffusers support for the inpainting model * fix debug_image to not crash with non-RGB images. * inpainting for the normal model [WIP] This seems to be performing well until the LAST STEP, at which point it dissolves to confetti. * fix off-by-one bug in cross-attention-control (#1774) prompt token sequences begin with a "beginning-of-sequence" marker <bos> and end with a repeated "end-of-sequence" marker <eos> - to make a default prompt length of <bos> + 75 prompt tokens + <eos>. the .swap() code was failing to take the column for <bos> at index 0 into account. the changes here do that, and also add extra handling for a single <eos> (which may be redundant but which is included for completeness). based on my understanding and some assumptions about how this all works, the reason .swap() nevertheless seemed to do the right thing, to some extent, is because over multiple steps the conditioning process in Stable Diffusion operates as a feedback loop. a change to token n-1 has flow-on effects to how the [1x4x64x64] latent tensor is modified by all the tokens after it, - and as the next step is processed, all the tokens before it as well. intuitively, a token's conditioning effects "echo" throughout the whole length of the prompt. so even though the token at n-1 was being edited when what the user actually wanted was to edit the token at n, it nevertheless still had some non-negligible effect, in roughly the right direction, often enough that it seemed like it was working properly. * refactor common CrossAttention stuff into a mixin so that the old ldm code can still work if necessary * inpainting for the normal model. I think it works this time. * diffusers: reset num_vectors_per_token sync with 44a00555718f1df173c60da0ed646cf700e29537 * diffusers: txt2img2img (hires_fix) with so much slicing and dicing of pipeline methods to stitch them together * refactor(diffusers): reduce some code duplication amongst the different tasks * fixup! refactor(diffusers): reduce some code duplication amongst the different tasks * diffusers: enable DPMSolver++ scheduler * diffusers: upgrade to diffusers 0.10, add Heun scheduler * diffusers(ModelCache): stopgap to make from_cpu compatible with diffusers * CI: default to diffusers-1.5 now that runwayml token requirement is gone * diffusers: update to 0.10 (and transformers to 4.25) * diffusers: use xformers when available diffusers no longer auto-enables this as of 0.10.2. * diffusers: make masked img2img behave better with multi-step schedulers re-randomizing the noise each step was confusing them. * diffusers: work more better with more models. fixed relative path problem with local models. fixed models on hub not always having a `fp16` branch. * diffusers: stopgap fix for attention_maps_callback crash after recent merge * fixup import merge conflicts correction for 061c5369a2247c6c92cd69606bcf54c4f1962a0b * test: add tests/inpainting inputs for masked img2img * diffusers(AddsMaskedGuidance): partial fix for k-schedulers Prevents them from crashing, but results are still hot garbage. * fix --safety_checker arg parsing and add note to diffusers loader about where safety checker gets called * generate: fix import error * CI: don't try to read the old init location * diffusers: support loading an alternate VAE * CI: remove sh-syntax if-statement so it doesn't crash powershell * CI: fold strings in yaml because backslash is not line-continuation in powershell * attention maps callback stuff for diffusers * build: fix syntax error in environment-mac * diffusers: add INITIAL_MODELS with diffusers-compatible repos * re-enable the embedding manager; closes #1778 * Squashed commit of the following: commit e4a956abc37fcb5cf188388b76b617bc5c8fda7d Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:43:07 2022 +0100 import new load handling from EmbeddingManager and cleanup commit c4abe91a5ba0d415b45bf734068385668b7a66e6 Merge: 032e856e 1efc6397 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:09:53 2022 +0100 Merge branch 'feature_textual_inversion_mgr' into dev/diffusers_with_textual_inversion_manager commit 032e856eefb3bbc39534f5daafd25764bcfcef8b Merge: 8b4f0fe9 bc515e24 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:08:01 2022 +0100 Merge remote-tracking branch 'upstream/dev/diffusers' into dev/diffusers_with_textual_inversion_manager commit 1efc6397fc6e61c1aff4b0258b93089d61de5955 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 15:04:28 2022 +0100 cleanup and add performance notes commit e400f804ac471a0ca2ba432fd658778b20c7bdab Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 14:45:07 2022 +0100 fix bug and update unit tests commit deb9ae0ae1016750e93ce8275734061f7285a231 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 14:28:29 2022 +0100 textual inversion manager seems to work commit 162e02505dec777e91a983c4d0fb52e950d25ff0 Merge: cbad4583 12769b3d Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:58:03 2022 +0100 Merge branch 'main' into feature_textual_inversion_mgr commit cbad45836c6aace6871a90f2621a953f49433131 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:54:10 2022 +0100 use position embeddings commit 070344c69b0e0db340a183857d0a787b348681d3 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:53:47 2022 +0100 Don't crash CLI on exceptions commit b035ac8c6772dfd9ba41b8eeb9103181cda028f8 Author: Damian Stewart <d@damianstewart.com> Date: Sun Dec 18 11:11:55 2022 +0100 add missing position_embeddings commit 12769b3d3562ef71e0f54946b532ad077e10043c Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 13:33:25 2022 +0100 debugging why it don't work commit bafb7215eabe1515ca5e8388fd3bb2f3ac5362cf Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 13:21:33 2022 +0100 debugging why it don't work commit 664a6e9e146b42d96703f0cc8baf8f5efec04ee1 Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 12:48:38 2022 +0100 use TextualInversionManager in place of embeddings (wip, doesn't work) commit 8b4f0fe9d6e4e2643b36dfa27864294785d7ba4e Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 12:48:38 2022 +0100 use TextualInversionManager in place of embeddings (wip, doesn't work) commit ffbe1ab11163ba712e353d89404e301d0e0c6cdf Merge: 6e4dad60 023df37e Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:37:31 2022 +0100 Merge branch 'feature_textual_inversion_mgr' into dev/diffusers commit 023df37efffa67434f77def7fc3c9dfb29f699fd Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:36:54 2022 +0100 cleanup commit 05fac594eaf79d0058e3c48deee93df603f136c2 Author: Damian Stewart <d@damianstewart.com> Date: Fri Dec 16 02:07:49 2022 +0100 tweak error checking commit 009f32ed39a7280997c3ffab112adadee0b44279 Author: damian <null@damianstewart.com> Date: Thu Dec 15 21:29:47 2022 +0100 unit tests passing for embeddings with vector length >1 commit beb1b08d9a98112ed2fe073580568e1a18698da3 Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 13:39:09 2022 +0100 more explicit equality tests when overwriting commit 44d8a5a7c85cdabc9ce3a54fd0769a10597b3ca9 Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 13:30:13 2022 +0100 wip textual inversion manager (unit tests passing for 1v embedding overwriting) commit 417c2b57d90924a839616bfb66804faab8039e4c Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 12:30:55 2022 +0100 wip textual inversion manager (unit tests passing for base stuff + padding) commit 2e80872e3b6f7fd7d8eb8928822bd824b63cb2ff Author: Damian Stewart <d@damianstewart.com> Date: Thu Dec 15 10:57:57 2022 +0100 wip new TextualInversionManager * stop using WeightedFrozenCLIPEmbedder * store diffusion models locally - configure_invokeai.py reconfigured to store diffusion models rather than CompVis models - hugging face caching model is used, but cache is set to ~/invokeai/models/repo_id - models.yaml does **NOT** use path, just repo_id - "repo_name" changed to "repo_id" to following hugging face conventions - Models are loaded with full precision pending further work. * allow non-local files during development * path takes priority over repo_id * MVP for model_cache and configure_invokeai - Feature complete (almost) - configure_invokeai.py downloads both .ckpt and diffuser models, along with their VAEs. Both types of download are controlled by a unified INITIAL_MODELS.yaml file. - model_cache can load both type of model and switches back and forth in CPU. No memory leaks detected TO DO: 1. I have not yet turned on the LocalOnly flag for diffuser models, so the code will check the Hugging Face repo for updates before using the locally cached models. This will break firewalled systems. I am thinking of putting in a global check for internet connectivity at startup time and setting the LocalOnly flag based on this. It would be good to check updates if there is connectivity. 2. I have not gone completely through INITIAL_MODELS.yaml to check which models are available as diffusers and which are not. So models like PaperCut and VoxelArt may not load properly. The runway and stability models are checked, as well as the Trinart models. 3. Add stanzas for SD 2.0 and 2.1 in INITIAL_MODELS.yaml REMAINING PROBLEMS NOT DIRECTLY RELATED TO MODEL_CACHE: 1. When loading a .ckpt file there are lots of messages like this: Warning! ldm.modules.attention.CrossAttention is no longer being maintained. Please use InvokeAICrossAttention instead. I'm not sure how to address this. 2. The ckpt models ***don't actually run*** due to the lack of special-case support for them in the generator objects. For example, here's the hard crash you get when you run txt2img against the legacy waifu-diffusion-1.3 model: ``` >> An error occurred: Traceback (most recent call last): File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 140, in main main_loop(gen, opt) File "/data/lstein/InvokeAI/ldm/invoke/CLI.py", line 371, in main_loop gen.prompt2image( File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image results = generator.generate( File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate image = make_image(x_T) File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image pipeline_output = pipeline.image_from_embeddings( File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1265, in __getattr__ raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'LatentDiffusion' object has no attribute 'image_from_embeddings' ``` 3. The inpainting diffusion model isn't working. Here's the output of "banana sushi" when inpainting-1.5 is loaded: ``` Traceback (most recent call last): File "/data/lstein/InvokeAI/ldm/generate.py", line 496, in prompt2image results = generator.generate( File "/data/lstein/InvokeAI/ldm/invoke/generator/base.py", line 108, in generate image = make_image(x_T) File "/data/lstein/InvokeAI/ldm/invoke/generator/txt2img.py", line 33, in make_image pipeline_output = pipeline.image_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 301, in image_from_embeddings result_latents, result_attention_map_saver = self.latents_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 330, in latents_from_embeddings result: PipelineIntermediateState = infer_latents_from_embeddings( File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 185, in __call__ for result in self.generator_method(*args, **kwargs): File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 367, in generate_latents_from_embeddings step_output = self.step(batched_t, latents, guidance_scale, File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/data/lstein/InvokeAI/ldm/invoke/generator/diffusers_pipeline.py", line 409, in step step_output = self.scheduler.step(noise_pred, timestep, latents, **extra_step_kwargs) File "/home/lstein/invokeai/.venv/lib/python3.9/site-packages/diffusers/schedulers/scheduling_lms_discrete.py", line 223, in step pred_original_sample = sample - sigma * model_output RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1 ``` * proper support for float32/float16 - configure script now correctly detects user's preference for fp16/32 and downloads the correct diffuser version. If fp16 version not available, falls back to fp32 version. - misc code cleanup and simplification in model_cache * add on-the-fly conversion of .ckpt to diffusers models 1. On-the-fly conversion code can be found in the file ldm/invoke/ckpt_to_diffusers.py. 2. A new !optimize command has been added to the CLI. Should be ported to Web GUI. User experience on the CLI is this: ``` invoke> !optimize /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt INFO: Converting legacy weights file /home/lstein/invokeai/models/ldm/stable-diffusion-v1/sd-v1-4.ckpt to optimized diffuser model. This operation will take 30-60s to complete. Success. Optimized model is now located at /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 Writing new config file entry for sd-v1-4... >> New configuration: sd-v1-4: description: Optimized version of sd-v1-4 format: diffusers path: /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 OK to import [n]? y >> Verifying that new model loads... >> Current VRAM usage: 2.60G >> Offloading stable-diffusion-2.1 to CPU >> Loading diffusers model from /home/lstein/tmp/invokeai/models/optimized-ckpts/sd-v1-4 | Using faster float16 precision You have disabled the safety checker for <class 'ldm.invoke.generator.diffusers_pipeline.StableDiffusionGeneratorPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion \ license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances,\ disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 . | training width x height = (512 x 512) >> Model loaded in 3.48s >> Max VRAM used to load the model: 2.17G >> Current VRAM usage:2.17G >> Textual inversions available: >> Setting Sampler to k_lms (LMSDiscreteScheduler) Keep model loaded? [y] ``` * add parallel set of generator files for ckpt legacy generation * generation using legacy ckpt models now working * diffusers: fix missing attention_maps_callback fix for 23eb80b40421b2bb8f4b6d3dd30490d11c447b36 * associate legacy CrossAttention with .ckpt models * enable autoconvert New --autoconvert CLI option will scan a designated directory for new .ckpt files, convert them into diffuser models, and import them into models.yaml. Works like this: invoke.py --autoconvert /path/to/weights/directory In ModelCache added two new methods: autoconvert_weights(config_path, weights_directory_path, models_directory_path) convert_and_import(ckpt_path, diffuser_path) * diffusers: update to diffusers 0.11 (from 0.10.2) * fix vae loading & width/height calculation * refactor: encapsulate these conditioning data into one container * diffusers: fix some noise-scaling issues by pushing the noise-mixing down to the common function * add support for safetensors and accelerate * set local_files_only when internet unreachable * diffusers: fix error-handling path when model repo has no fp16 branch * fix generatorinpaint error Fixes : "ModuleNotFoundError: No module named 'ldm.invoke.generatorinpaint' https://github.com/invoke-ai/InvokeAI/pull/1583#issuecomment-1363634318 * quench diffuser safety-checker warning * diffusers: support stochastic DDIM eta parameter * fix conda env creation on macos * fix cross-attention with diffusers 0.11 * diffusers: the VAE needs to be tiling as well as the U-Net * diffusers: comment on subfolders * diffusers: embiggen! * diffusers: make model_cache.list_models serializable * diffusers(inpaint): restore scaling functionality * fix requirements clash between numba and numpy 1.24 * diffusers: allow inpainting model to do non-inpainting tasks * start expanding model_cache functionality * add import_ckpt_model() and import_diffuser_model() methods to model_manager - in addition, model_cache.py is now renamed to model_manager.py * allow "recommended" flag to be optional in INITIAL_MODELS.yaml * configure_invokeai now downloads VAE diffusers in advance * rename ModelCache to ModelManager * remove support for `repo_name` in models.yaml * check for and refuse to load embeddings trained on incompatible models * models.yaml.example: s/repo_name/repo_id and remove extra INITIAL_MODELS now that the main one has diffusers models in it. * add MVP textual inversion script * refactor(InvokeAIDiffuserComponent): factor out _combine() * InvokeAIDiffuserComponent: implement threshold * InvokeAIDiffuserComponent: diagnostic logs for threshold ...this does not look right * add a curses-based frontend to textual inversion - not quite working yet - requires npyscreen installed - on windows will also have the windows-curses requirement, but not added to requirements yet * add curses-based interface for textual inversion * fix crash in convert_and_import() - This corrects a "local variable referenced before assignment" error in model_manager.convert_and_import() * potential workaround for no 'state_dict' key error - As reported in https://github.com/huggingface/diffusers/issues/1876 * create TI output dir if needed * Update environment-lin-cuda.yml (#2159) Fixing line 42 to be the proper order to define the transformers requirement: ~= instead of =~ * diffusers: update sampler-to-scheduler mapping based on https://github.com/huggingface/diffusers/issues/277#issuecomment-1371428672 * improve user exp for ckt to diffusers conversion - !optimize_models command now operates on an existing ckpt file entry in models.yaml - replaces existing entry, rather than adding a new one - offers to delete the ckpt file after conversion * web: adapt progress callback to deal with old generator or new diffusers pipeline * clean-up model_manager code - add_model() verified to work for .ckpt local paths, .ckpt remote URLs, diffusers local paths, and diffusers repo_ids - convert_and_import() verified to work for local and remove .ckpt files * handle edge cases for import_model() and convert_model() * add support for safetensor .ckpt files * fix name error * code cleanup with pyflake * improve model setting behavior - If the user enters an invalid model name at startup time, will not try to load it, warn, and use default model - CLI UI enhancement: include currently active model in the command line prompt. * update test-invoke-pip.yml - fix model cache path to point to runwayml/stable-diffusion-v1-5 - remove `skip-sd-weights` from configure_invokeai.py args * exclude dev/diffusers from "fail for draft PRs" * disable "fail on PR jobs" * re-add `--skip-sd-weights` since no space * update workflow environments - include `INVOKE_MODEL_RECONFIGURE: '--yes'` * clean up model load failure handling - Allow CLI to run even when no model is defined or loadable. - Inhibit stack trace when model load fails - only show last error - Give user *option* to run configure_invokeai.py when no models successfully load. - Restart invokeai after reconfiguration. * further edge-case handling 1) only one model in models.yaml file, and that model is broken 2) no models in models.yaml 3) models.yaml doesn't exist at all * fix incorrect model status listing - "cached" was not being returned from list_models() - normalize handling of exceptions during model loading: - Passing an invalid model name to generate.set_model() will return a KeyError - All other exceptions are returned as the appropriate Exception * CI: do download weights (if not already cached) * diffusers: fix scheduler loading in offline mode * CI: fix model name (no longer has `diffusers-` prefix) * Update txt2img2img.py (#2256) * fixes to share models with HuggingFace cache system - If HF_HOME environment variable is defined, then all huggingface models are stored in that directory following the standard conventions. - For seamless interoperability, set HF_HOME to ~/.cache/huggingface - If HF_HOME not defined, then models are stored in ~/invokeai/models. This is equivalent to setting HF_HOME to ~/invokeai/models A future commit will add a migration mechanism so that this change doesn't break previous installs. * feat - make model storage compatible with hugging face caching system This commit alters the InvokeAI model directory to be compatible with hugging face, making it easier to share diffusers (and other models) across different programs. - If the HF_HOME environment variable is not set, then models are cached in ~/invokeai/models in a format that is identical to the HuggingFace cache. - If HF_HOME is set, then models are cached wherever HF_HOME points. - To enable sharing with other HuggingFace library clients, set HF_HOME to ~/.cache/huggingface to set the default cache location or to ~/invokeai/models to have huggingface cache inside InvokeAI. * fixes to share models with HuggingFace cache system - If HF_HOME environment variable is defined, then all huggingface models are stored in that directory following the standard conventions. - For seamless interoperability, set HF_HOME to ~/.cache/huggingface - If HF_HOME not defined, then models are stored in ~/invokeai/models. This is equivalent to setting HF_HOME to ~/invokeai/models A future commit will add a migration mechanism so that this change doesn't break previous installs. * fix error "no attribute CkptInpaint" * model_manager.list_models() returns entire model config stanza+status * Initial Draft - Model Manager Diffusers * added hash function to diffusers * implement sha256 hashes on diffusers models * Add Model Manager Support for Diffusers * fix various problems with model manager - in cli import functions, fix not enough values to unpack from _get_name_and_desc() - fix crash when using old-style vae: value with new-style diffuser * rebuild frontend * fix dictconfig-not-serializable issue * fix NoneType' object is not subscriptable crash in model_manager * fix "str has no attribute get" error in model_manager list_models() * Add path and repo_id support for Diffusers Model Manager Also fixes bugs * Fix tooltip IT localization not working * Add Version Number To WebUI * Optimize Model Search * Fix incorrect font on the Model Manager UI * Fix image degradation on merge fixes - [Experimental] This change should effectively fix a couple of things. - Fix image degradation on subsequent merges of the canvas layers. - Fix the slight transparent border that is left behind when filling the bounding box with a color. - Fix the left over line of color when filling a bounding box with color. So far there are no side effects for this. If any, please report. * Add local model filtering for Diffusers / Checkpoints * Go to home on modal close for the Add Modal UI * Styling Fixes * Model Manager Diffusers Localization Update * Add Safe Tensor scanning to Model Manager * Fix model edit form dispatching string values instead of numbers. * Resolve VAE handling / edge cases for supplied repos * defer injecting tokens for textual inversions until they're used for the first time * squash a console warning * implement model migration check * add_model() overwrites previous config rather than merges * fix model config file attribute merging * fix precision handling in textual inversion script * allow ckpt conversion script to work with safetensors .ckpts Applied patch here: https://github.com/huggingface/diffusers/commit/beb932c5d111872c5e45387e7b1b2b3dd0524a47 * fix name "args" is not defined crash in textual_inversion_training * fix a second NameError: name 'args' is not defined crash * fix loading of the safety checker from the global cache dir * add installation step to textual inversion frontend - After a successful training run, the script will copy learned_embeds.bin to a subfolder of the embeddings directory. - User given the option to delete the logs and intermediate checkpoints (which together use 7-8G of space) - If textual inversion training fails, reports the error gracefully. * don't crash out on incompatible embeddings - put try: blocks around places where the system tries to load an embedding which is incompatible with the currently loaded model * add support for checkpoint resuming * textual inversion preferences are saved and restored between sessions - Preferences are stored in a file named text-inversion-training/preferences.conf - Currently the resume-from-checkpoint option is not working correctly. Possible bug in textual_inversion_training.py? * copy learned_embeddings.bin into right location * add front end for diffusers model merging - Front end doesn't do anything yet!!!! - Made change to model name parsing in CLI to support ability to have merged models with the "+" character in their names. * improve inpainting experience - recommend ckpt version of inpainting-1.5 to user - fix get_noise() bug in ckpt version of omnibus.py * update environment*yml * tweak instructions to install HuggingFace token * bump version number * enhance update scripts - update scripts will now fetch new INITIAL_MODELS.yaml so that configure_invokeai.py will know about the diffusers versions. * enhance invoke.sh/invoke.bat launchers - added configure_invokeai.py to menu - menu defaults to browser-based invoke * remove conda workflow (#2321) * fix `token_ids has shape torch.Size([79]) - expected [77]` * update CHANGELOG.md with 2.3.* info - Add information on how formats have changed and the upgrade process. - Add short bug list. Co-authored-by: Damian Stewart <d@damianstewart.com> Co-authored-by: Damian Stewart <null@damianstewart.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Wybartel-luxmc <37852506+Wybartel-luxmc@users.noreply.github.com> Co-authored-by: mauwii <Mauwii@outlook.de> Co-authored-by: mickr777 <115216705+mickr777@users.noreply.github.com> Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com> Co-authored-by: Eugene Brodsky <ebr@users.noreply.github.com> Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com>
2023-01-15 14:22:46 +00:00
params:
freeze: True
layer: "penultimate"
Reference in New Issue Copy Permalink