From be4f3fa5c611fcab5b346ab8989be9ba95328330 Mon Sep 17 00:00:00 2001 From: Millun Atluri Date: Thu, 16 Nov 2023 16:32:55 +1100 Subject: [PATCH] Added LCM-LoRA --- docs/features/LORAS.md | 51 +++++++++++++++++++ .../{CONCEPTS.md => TEXTUAL_INVERSIONS.md} | 36 +------------ mkdocs.yml | 7 ++- 3 files changed, 57 insertions(+), 37 deletions(-) create mode 100644 docs/features/LORAS.md rename docs/features/{CONCEPTS.md => TEXTUAL_INVERSIONS.md} (66%) diff --git a/docs/features/LORAS.md b/docs/features/LORAS.md new file mode 100644 index 0000000000..3426311b41 --- /dev/null +++ b/docs/features/LORAS.md @@ -0,0 +1,51 @@ +--- +title: LoRAs & LCM-LoRAs +--- + +# :material-library-shelves: LoRAs & LCM-LoRAs + +With the advances in research, many new capabilities are available to customize the knowledge and understanding of novel concepts not originally contained in the base model. + +## LoRAs + +Low-Rank Adaptation (LoRA) files are models that customize the output of Stable Diffusion +image generation. Larger than embeddings, but much smaller than full +models, they augment SD with improved understanding of subjects and +artistic styles. + +Unlike TI files, LoRAs do not introduce novel vocabulary into the +model's known tokens. Instead, LoRAs augment the model's weights that +are applied to generate imagery. LoRAs may be supplied with a +"trigger" word that they have been explicitly trained on, or may +simply apply their effect without being triggered. + +LoRAs are typically stored in .safetensors files, which are the most +secure way to store and transmit these types of weights. You may +install any number of `.safetensors` LoRA files simply by copying them +into the `autoimport/lora` directory of the corresponding InvokeAI models +directory (usually `invokeai` in your home directory). + +To use these when generating, open the LoRA menu item in the options +panel, select the LoRAs you want to apply and ensure that they have +the appropriate weight recommended by the model provider. Typically, +most LoRAs perform best at a weight of .75-1. + + +## LCM-LoRAs +Latent Consistency Models (LCMs) allowed a reduced number of steps to be used to generate images with Stable Diffusion. These are created by distilling base models, creating models that only require a small number of steps to generate images. However, LCMs require that any fine-tune of a base model be distilled to be used as an LCM. + +LCM-LoRAs are models that provide the benefit of LCMs but are able to be used as LoRAs and applied to any fine tune of a base model. LCM-LoRAs are created by training a small number of adapters, rather than distilling the entire fine-tuned base model. The resulting LoRA can be used the same way as a standard LoRA, but with a greatly reduced step count. This enables SDXL images to be generated up to 10x faster than without the use of LCM-LoRAs. + + +**Using LCM-LoRAs** +LCM-LoRAs are natively supported in InvokeAI throughout the application. To get started, install any diffusers format LCM-LoRAs using the model manager and select it in the LoRA field. + +There are a number parameter differences when using LCM-LoRAs and standard generation: +- When using LCM-LoRAs, the LoRA strength should be lower than if using a standard LoRA, with 0.35 recommended as a starting point. +- The LCM scheduler should be used for generation +- CFG-Scale should be reduced to ~1 +- Steps should be reduced in the range of 4-8 + +Standard LoRAs can also be used alongside LCM-LoRAs, but will also require a lower strength, with 0.45 being recommended as a starting point. + +More information can be found here: https://huggingface.co/blog/lcm_lora#fast-inference-with-sdxl-lcm-loras diff --git a/docs/features/CONCEPTS.md b/docs/features/TEXTUAL_INVERSIONS.md similarity index 66% rename from docs/features/CONCEPTS.md rename to docs/features/TEXTUAL_INVERSIONS.md index 5f3d2d961f..a3ede80d1f 100644 --- a/docs/features/CONCEPTS.md +++ b/docs/features/TEXTUAL_INVERSIONS.md @@ -1,12 +1,3 @@ ---- -title: Textual Inversion Embeddings and LoRAs ---- - -# :material-library-shelves: Textual Inversions and LoRAs - -With the advances in research, many new capabilities are available to customize the knowledge and understanding of novel concepts not originally contained in the base model. - - ## Using Textual Inversion Files Textual inversion (TI) files are small models that customize the output of @@ -61,29 +52,4 @@ files it finds there for compatible models. At startup you will see a message si >> Current embedding manager terms: , ``` To use these when generating, simply type the `<` key in your prompt to open the Textual Inversion WebUI and -select the embedding you'd like to use. This UI has type-ahead support, so you can easily find supported embeddings. - -## Using LoRAs - -LoRA files are models that customize the output of Stable Diffusion -image generation. Larger than embeddings, but much smaller than full -models, they augment SD with improved understanding of subjects and -artistic styles. - -Unlike TI files, LoRAs do not introduce novel vocabulary into the -model's known tokens. Instead, LoRAs augment the model's weights that -are applied to generate imagery. LoRAs may be supplied with a -"trigger" word that they have been explicitly trained on, or may -simply apply their effect without being triggered. - -LoRAs are typically stored in .safetensors files, which are the most -secure way to store and transmit these types of weights. You may -install any number of `.safetensors` LoRA files simply by copying them -into the `autoimport/lora` directory of the corresponding InvokeAI models -directory (usually `invokeai` in your home directory). - -To use these when generating, open the LoRA menu item in the options -panel, select the LoRAs you want to apply and ensure that they have -the appropriate weight recommended by the model provider. Typically, -most LoRAs perform best at a weight of .75-1. - +select the embedding you'd like to use. This UI has type-ahead support, so you can easily find supported embeddings. \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 6f08220e6d..de08d4ff4a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -144,12 +144,15 @@ nav: - Control Adapters: 'features/CONTROLNET.md' - Image-to-Image: 'features/IMG2IMG.md' - Controlling Logging: 'features/LOGGING.md' + - LoRAs & LCM-LoRAs: 'features/LORAS.md' - Model Merging: 'features/MODEL_MERGING.md' - - Using Nodes : 'nodes/overview.md' + - Nodes & Workflows: 'nodes/overview.md' - NSFW Checker: 'features/WATERMARK+NSFW.md' - Postprocessing: 'features/POSTPROCESS.md' - Prompting Features: 'features/PROMPTS.md' - - Textual Inversion Training: 'features/TRAINING.md' + - Textual Inversions: + - Textual Inversions: 'features/TEXTUAL_INVERSIONS.md' + - Textual Inversion Training: 'features/TRAINING.md' - Unified Canvas: 'features/UNIFIED_CANVAS.md' - InvokeAI Web Server: 'features/WEB.md' - WebUI Hotkeys: "features/WEBUIHOTKEYS.md"