diff --git a/docs/assets/gallery/board_settings.png b/docs/assets/gallery/board_settings.png new file mode 100644 index 0000000000..44c4ef240b Binary files /dev/null and b/docs/assets/gallery/board_settings.png differ diff --git a/docs/assets/gallery/board_tabs.png b/docs/assets/gallery/board_tabs.png new file mode 100644 index 0000000000..23e5f8a91c Binary files /dev/null and b/docs/assets/gallery/board_tabs.png differ diff --git a/docs/assets/gallery/board_thumbnails.png b/docs/assets/gallery/board_thumbnails.png new file mode 100644 index 0000000000..1c739d4854 Binary files /dev/null and b/docs/assets/gallery/board_thumbnails.png differ diff --git a/docs/assets/gallery/gallery.png b/docs/assets/gallery/gallery.png new file mode 100644 index 0000000000..89f2dd1b46 Binary files /dev/null and b/docs/assets/gallery/gallery.png differ diff --git a/docs/assets/gallery/image_menu.png b/docs/assets/gallery/image_menu.png new file mode 100644 index 0000000000..2f10f280ac Binary files /dev/null and b/docs/assets/gallery/image_menu.png differ diff --git a/docs/assets/gallery/info_button.png b/docs/assets/gallery/info_button.png new file mode 100644 index 0000000000..539cd6252e Binary files /dev/null and b/docs/assets/gallery/info_button.png differ diff --git a/docs/assets/gallery/thumbnail_menu.png b/docs/assets/gallery/thumbnail_menu.png new file mode 100644 index 0000000000..a56caadbd8 Binary files /dev/null and b/docs/assets/gallery/thumbnail_menu.png differ diff --git a/docs/assets/gallery/top_controls.png b/docs/assets/gallery/top_controls.png new file mode 100644 index 0000000000..c5d3cdc854 Binary files /dev/null and b/docs/assets/gallery/top_controls.png differ diff --git a/docs/features/GALLERY.md b/docs/features/GALLERY.md new file mode 100644 index 0000000000..cc84dbf704 --- /dev/null +++ b/docs/features/GALLERY.md @@ -0,0 +1,92 @@ +--- +title: InvokeAI Gallery Panel +--- + +# :material-web: InvokeAI Gallery Panel + +## Quick guided walkthrough of the Gallery Panel's features + +The Gallery Panel is a fast way to review, find, and make use of images you've +generated and loaded. The Gallery is divided into Boards. The Uncategorized board is always +present but you can create your own for better organization. + +![image](../assets/gallery/gallery.png) + +### Board Display and Settings + +At the very top of the Gallery Panel are the boards disclosure and settings buttons. + +![image](../assets/gallery/top_controls.png) + +The disclosure button shows the name of the currently selected board and allows you to show and hide the board thumbnails (shown in the image below). + +![image](../assets/gallery/board_thumbnails.png) + +The settings button opens a list of options. + +![image](../assets/gallery/board_settings.png) + +- ***Image Size*** this slider lets you control the size of the image previews (images of three different sizes). +- ***Auto-Switch to New Images*** if you turn this on, whenever a new image is generated, it will automatically be loaded into the current image panel on the Text to Image tab and into the result panel on the [Image to Image](IMG2IMG.md) tab. This will happen invisibly if you are on any other tab when the image is generated. +- ***Auto-Assign Board on Click*** whenever an image is generated or saved, it always gets put in a board. The board it gets put into is marked with AUTO (image of board marked). Turning on Auto-Assign Board on Click will make whichever board you last selected be the destination when you click Invoke. That means you can click Invoke, select a different board, and then click Invoke again and the two images will be put in two different boards. (bold)It's the board selected when Invoke is clicked that's used, not the board that's selected when the image is finished generating.(bold) Turning this off, enables the Auto-Add Board drop down which lets you set one specific board to always put generated images into. This also enables and disables the Auto-add to this Board menu item described below. +- ***Always Show Image Size Badge*** this toggles whether to show image sizes for each image preview (show two images, one with sizes shown, one without) + +Below these two buttons, you'll see the Search Boards text entry area. You use this to search for specific boards by the name of the board. +Next to it is the Add Board (+) button which lets you add new boards. Boards can be renamed by clicking on the name of the board under its thumbnail and typing in the new name. + +### Board Thumbnail Menu + +Each board has a context menu (ctrl+click / right-click). + +![image](../assets/gallery/thumbnail_menu.png) + +- ***Auto-add to this Board*** if you've disabled Auto-Assign Board on Click in the board settings, you can use this option to set this board to be where new images are put. +- ***Download Board*** this will add all the images in the board into a zip file and provide a link to it in a notification (image of notification) +- ***Delete Board*** this will delete the board +> [!CAUTION] +> This will delete all the images in the board and the board itself. + +### Board Contents + +Every board is organized by two tabs, Images and Assets. + +![image](../assets/gallery/board_tabs.png) + +Images are the Invoke-generated images that are placed into the board. Assets are images that you upload into Invoke to be used as an [Image Prompt](https://support.invoke.ai/support/solutions/articles/151000159340-using-the-image-prompt-adapter-ip-adapter-) or in the [Image to Image](IMG2IMG.md) tab. + +### Image Thumbnail Menu + +Every image generated by Invoke has its generation information stored as text inside the image file itself. This can be read directly by selecting the image and clicking on the Info button ![image](../assets/gallery/info_button.png) in any of the image result panels. + +Each image also has a context menu (ctrl+click / right-click). + +![image](../assets/gallery/image_menu.png) + + The options are (items marked with an * will not work with images that lack generation information): +- ***Open in New Tab*** this will open the image alone in a new browser tab, separate from the Invoke interface. +- ***Download Image*** this will trigger your browser to download the image. +- ***Load Workflow **** this will load any workflow settings into the Workflow tab and automatically open it. +- ***Remix Image **** this will load all of the image's generation information, (bold)excluding its Seed, into the left hand control panel +- ***Use Prompt **** this will load only the image's text prompts into the left-hand control panel +- ***Use Seed **** this will load only the image's Seed into the left-hand control panel +- ***Use All **** this will load all of the image's generation information into the left-hand control panel +- ***Send to Image to Image*** this will put the image into the left-hand panel in the Image to Image tab ana automatically open it +- ***Send to Unified Canvas*** This will (bold)replace whatever is already present(bold) in the Unified Canvas tab with the image and automatically open the tab +- ***Change Board*** this will oipen a small window that will let you move the image to a different board. This is the same as dragging the image to that board's thumbnail. +- ***Star Image*** this will add the image to the board's list of starred images that are always kept at the top of the gallery. This is the same as clicking on the star on the top right-hand side of the image that appears when you hover over the image with the mouse +- ***Delete Image*** this will delete the image from the board +> [!CAUTION] +> This will delete the image entirely from Invoke. + +## Summary + +This walkthrough only covers the Gallery interface and Boards. Actually generating images is handled by [Prompts](PROMPTS.md), the [Image to Image](IMG2IMG.md) tab, and the [Unified Canvas](UNIFIED_CANVAS.md). + +## Acknowledgements + +A huge shout-out to the core team working to make the Web GUI a reality, +including [psychedelicious](https://github.com/psychedelicious), +[Kyle0654](https://github.com/Kyle0654) and +[blessedcoolant](https://github.com/blessedcoolant). +[hipsterusername](https://github.com/hipsterusername) was the team's unofficial +cheerleader and added tooltips/docs. diff --git a/docs/features/WEB.md b/docs/features/WEB.md index 765a66aeac..15a489b932 100644 --- a/docs/features/WEB.md +++ b/docs/features/WEB.md @@ -54,7 +54,7 @@ main sections: of buttons at the top lets you modify and manipulate the image in various ways. -3. A **gallery** section on the left that contains a history of the images you +3. A **gallery** section on the right that contains a history of the images you have generated. These images are read and written to the directory specified in the `INVOKEAIROOT/invokeai.yaml` initialization file, usually a directory named `outputs` in `INVOKEAIROOT`. diff --git a/docs/installation/INSTALL_DEVELOPMENT.md b/docs/installation/INSTALL_DEVELOPMENT.md index 3f42846556..ead6b3bc8d 100644 --- a/docs/installation/INSTALL_DEVELOPMENT.md +++ b/docs/installation/INSTALL_DEVELOPMENT.md @@ -23,6 +23,7 @@ If you have an interest in how InvokeAI works, or you would like to add features 1. [Fork and clone] the [InvokeAI repo]. 1. Follow the [manual installation] docs to create a new virtual environment for the development install. + - Create a new folder outside the repo root for the installation and create the venv inside that folder. - When installing the InvokeAI package, add `-e` to the command so you get an [editable install]. 1. Install the [frontend dev toolchain] and do a production build of the UI as described. 1. You can now run the app as described in the [manual installation] docs. diff --git a/invokeai/app/api_app.py b/invokeai/app/api_app.py index 87eaefc020..ceaeb95147 100644 --- a/invokeai/app/api_app.py +++ b/invokeai/app/api_app.py @@ -28,7 +28,7 @@ from invokeai.app.api.no_cache_staticfiles import NoCacheStaticFiles from invokeai.app.invocations.model import ModelIdentifierField from invokeai.app.services.config.config_default import get_config from invokeai.app.services.session_processor.session_processor_common import ProgressImage -from invokeai.backend.util.devices import get_torch_device_name +from invokeai.backend.util.devices import TorchDevice from ..backend.util.logging import InvokeAILogger from .api.dependencies import ApiDependencies @@ -63,7 +63,7 @@ logger = InvokeAILogger.get_logger(config=app_config) mimetypes.add_type("application/javascript", ".js") mimetypes.add_type("text/css", ".css") -torch_device_name = get_torch_device_name() +torch_device_name = TorchDevice.get_torch_device_name() logger.info(f"Using torch device: {torch_device_name}") diff --git a/invokeai/app/invocations/compel.py b/invokeai/app/invocations/compel.py index 481a2d2e4b..158f11a58e 100644 --- a/invokeai/app/invocations/compel.py +++ b/invokeai/app/invocations/compel.py @@ -24,7 +24,7 @@ from invokeai.backend.stable_diffusion.diffusion.conditioning_data import ( ConditioningFieldData, SDXLConditioningInfo, ) -from invokeai.backend.util.devices import torch_dtype +from invokeai.backend.util.devices import TorchDevice from .baseinvocation import BaseInvocation, BaseInvocationOutput, invocation, invocation_output from .model import CLIPField @@ -99,7 +99,7 @@ class CompelInvocation(BaseInvocation): tokenizer=tokenizer, text_encoder=text_encoder, textual_inversion_manager=ti_manager, - dtype_for_device_getter=torch_dtype, + dtype_for_device_getter=TorchDevice.choose_torch_dtype, truncate_long_prompts=False, ) @@ -193,7 +193,7 @@ class SDXLPromptInvocationBase: tokenizer=tokenizer, text_encoder=text_encoder, textual_inversion_manager=ti_manager, - dtype_for_device_getter=torch_dtype, + dtype_for_device_getter=TorchDevice.choose_torch_dtype, truncate_long_prompts=False, # TODO: returned_embeddings_type=ReturnedEmbeddingsType.PENULTIMATE_HIDDEN_STATES_NON_NORMALIZED, # TODO: clip skip requires_pooled=get_pooled, diff --git a/invokeai/app/invocations/latent.py b/invokeai/app/invocations/latent.py index 85f1e2bf24..4534df89c1 100644 --- a/invokeai/app/invocations/latent.py +++ b/invokeai/app/invocations/latent.py @@ -72,15 +72,12 @@ from ...backend.stable_diffusion.diffusers_pipeline import ( image_resized_to_grid_as_tensor, ) from ...backend.stable_diffusion.schedulers import SCHEDULER_MAP -from ...backend.util.devices import choose_precision, choose_torch_device +from ...backend.util.devices import TorchDevice from .baseinvocation import BaseInvocation, BaseInvocationOutput, invocation, invocation_output from .controlnet_image_processors import ControlField from .model import ModelIdentifierField, UNetField, VAEField -if choose_torch_device() == torch.device("mps"): - from torch import mps - -DEFAULT_PRECISION = choose_precision(choose_torch_device()) +DEFAULT_PRECISION = TorchDevice.choose_torch_dtype() @invocation_output("scheduler_output") @@ -960,9 +957,7 @@ class DenoiseLatentsInvocation(BaseInvocation): # https://discuss.huggingface.co/t/memory-usage-by-later-pipeline-stages/23699 result_latents = result_latents.to("cpu") - torch.cuda.empty_cache() - if choose_torch_device() == torch.device("mps"): - mps.empty_cache() + TorchDevice.empty_cache() name = context.tensors.save(tensor=result_latents) return LatentsOutput.build(latents_name=name, latents=result_latents, seed=None) @@ -1029,9 +1024,7 @@ class LatentsToImageInvocation(BaseInvocation, WithMetadata, WithBoard): vae.disable_tiling() # clear memory as vae decode can request a lot - torch.cuda.empty_cache() - if choose_torch_device() == torch.device("mps"): - mps.empty_cache() + TorchDevice.empty_cache() with torch.inference_mode(): # copied from diffusers pipeline @@ -1043,9 +1036,7 @@ class LatentsToImageInvocation(BaseInvocation, WithMetadata, WithBoard): image = VaeImageProcessor.numpy_to_pil(np_image)[0] - torch.cuda.empty_cache() - if choose_torch_device() == torch.device("mps"): - mps.empty_cache() + TorchDevice.empty_cache() image_dto = context.images.save(image=image) @@ -1084,9 +1075,7 @@ class ResizeLatentsInvocation(BaseInvocation): def invoke(self, context: InvocationContext) -> LatentsOutput: latents = context.tensors.load(self.latents.latents_name) - - # TODO: - device = choose_torch_device() + device = TorchDevice.choose_torch_device() resized_latents = torch.nn.functional.interpolate( latents.to(device), @@ -1097,9 +1086,8 @@ class ResizeLatentsInvocation(BaseInvocation): # https://discuss.huggingface.co/t/memory-usage-by-later-pipeline-stages/23699 resized_latents = resized_latents.to("cpu") - torch.cuda.empty_cache() - if device == torch.device("mps"): - mps.empty_cache() + + TorchDevice.empty_cache() name = context.tensors.save(tensor=resized_latents) return LatentsOutput.build(latents_name=name, latents=resized_latents, seed=self.latents.seed) @@ -1126,8 +1114,7 @@ class ScaleLatentsInvocation(BaseInvocation): def invoke(self, context: InvocationContext) -> LatentsOutput: latents = context.tensors.load(self.latents.latents_name) - # TODO: - device = choose_torch_device() + device = TorchDevice.choose_torch_device() # resizing resized_latents = torch.nn.functional.interpolate( @@ -1139,9 +1126,7 @@ class ScaleLatentsInvocation(BaseInvocation): # https://discuss.huggingface.co/t/memory-usage-by-later-pipeline-stages/23699 resized_latents = resized_latents.to("cpu") - torch.cuda.empty_cache() - if device == torch.device("mps"): - mps.empty_cache() + TorchDevice.empty_cache() name = context.tensors.save(tensor=resized_latents) return LatentsOutput.build(latents_name=name, latents=resized_latents, seed=self.latents.seed) @@ -1273,8 +1258,7 @@ class BlendLatentsInvocation(BaseInvocation): if latents_a.shape != latents_b.shape: raise Exception("Latents to blend must be the same size.") - # TODO: - device = choose_torch_device() + device = TorchDevice.choose_torch_device() def slerp( t: Union[float, npt.NDArray[Any]], # FIXME: maybe use np.float32 here? @@ -1327,9 +1311,8 @@ class BlendLatentsInvocation(BaseInvocation): # https://discuss.huggingface.co/t/memory-usage-by-later-pipeline-stages/23699 blended_latents = blended_latents.to("cpu") - torch.cuda.empty_cache() - if device == torch.device("mps"): - mps.empty_cache() + + TorchDevice.empty_cache() name = context.tensors.save(tensor=blended_latents) return LatentsOutput.build(latents_name=name, latents=blended_latents) diff --git a/invokeai/app/invocations/noise.py b/invokeai/app/invocations/noise.py index 6e5612b8b0..931e639106 100644 --- a/invokeai/app/invocations/noise.py +++ b/invokeai/app/invocations/noise.py @@ -9,7 +9,7 @@ from invokeai.app.invocations.fields import FieldDescriptions, InputField, Laten from invokeai.app.services.shared.invocation_context import InvocationContext from invokeai.app.util.misc import SEED_MAX -from ...backend.util.devices import choose_torch_device, torch_dtype +from ...backend.util.devices import TorchDevice from .baseinvocation import ( BaseInvocation, BaseInvocationOutput, @@ -46,7 +46,7 @@ def get_noise( height // downsampling_factor, width // downsampling_factor, ], - dtype=torch_dtype(device), + dtype=TorchDevice.choose_torch_dtype(device=device), device=noise_device_type, generator=generator, ).to("cpu") @@ -111,14 +111,14 @@ class NoiseInvocation(BaseInvocation): @field_validator("seed", mode="before") def modulo_seed(cls, v): - """Returns the seed modulo (SEED_MAX + 1) to ensure it is within the valid range.""" + """Return the seed modulo (SEED_MAX + 1) to ensure it is within the valid range.""" return v % (SEED_MAX + 1) def invoke(self, context: InvocationContext) -> NoiseOutput: noise = get_noise( width=self.width, height=self.height, - device=choose_torch_device(), + device=TorchDevice.choose_torch_device(), seed=self.seed, use_cpu=self.use_cpu, ) diff --git a/invokeai/app/invocations/upscale.py b/invokeai/app/invocations/upscale.py index d687384fcb..deaf5696c6 100644 --- a/invokeai/app/invocations/upscale.py +++ b/invokeai/app/invocations/upscale.py @@ -4,7 +4,6 @@ from typing import Literal import cv2 import numpy as np -import torch from PIL import Image from pydantic import ConfigDict @@ -14,7 +13,7 @@ from invokeai.app.services.shared.invocation_context import InvocationContext from invokeai.app.util.download_with_progress import download_with_progress_bar from invokeai.backend.image_util.basicsr.rrdbnet_arch import RRDBNet from invokeai.backend.image_util.realesrgan.realesrgan import RealESRGAN -from invokeai.backend.util.devices import choose_torch_device +from invokeai.backend.util.devices import TorchDevice from .baseinvocation import BaseInvocation, invocation from .fields import InputField, WithBoard, WithMetadata @@ -35,9 +34,6 @@ ESRGAN_MODEL_URLS: dict[str, str] = { "RealESRGAN_x2plus.pth": "https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.1/RealESRGAN_x2plus.pth", } -if choose_torch_device() == torch.device("mps"): - from torch import mps - @invocation("esrgan", title="Upscale (RealESRGAN)", tags=["esrgan", "upscale"], category="esrgan", version="1.3.2") class ESRGANInvocation(BaseInvocation, WithMetadata, WithBoard): @@ -120,9 +116,7 @@ class ESRGANInvocation(BaseInvocation, WithMetadata, WithBoard): upscaled_image = upscaler.upscale(cv2_image) pil_image = Image.fromarray(cv2.cvtColor(upscaled_image, cv2.COLOR_BGR2RGB)).convert("RGBA") - torch.cuda.empty_cache() - if choose_torch_device() == torch.device("mps"): - mps.empty_cache() + TorchDevice.empty_cache() image_dto = context.images.save(image=pil_image) diff --git a/invokeai/app/services/config/config_default.py b/invokeai/app/services/config/config_default.py index f453a56584..54a092d03e 100644 --- a/invokeai/app/services/config/config_default.py +++ b/invokeai/app/services/config/config_default.py @@ -27,12 +27,12 @@ DEFAULT_RAM_CACHE = 10.0 DEFAULT_VRAM_CACHE = 0.25 DEFAULT_CONVERT_CACHE = 20.0 DEVICE = Literal["auto", "cpu", "cuda", "cuda:1", "mps"] -PRECISION = Literal["auto", "float16", "bfloat16", "float32", "autocast"] +PRECISION = Literal["auto", "float16", "bfloat16", "float32"] ATTENTION_TYPE = Literal["auto", "normal", "xformers", "sliced", "torch-sdp"] ATTENTION_SLICE_SIZE = Literal["auto", "balanced", "max", 1, 2, 3, 4, 5, 6, 7, 8] LOG_FORMAT = Literal["plain", "color", "syslog", "legacy"] LOG_LEVEL = Literal["debug", "info", "warning", "error", "critical"] -CONFIG_SCHEMA_VERSION = "4.0.0" +CONFIG_SCHEMA_VERSION = "4.0.1" def get_default_ram_cache_size() -> float: @@ -105,7 +105,7 @@ class InvokeAIAppConfig(BaseSettings): lazy_offload: Keep models in VRAM until their space is needed. log_memory_usage: If True, a memory snapshot will be captured before and after every model cache operation, and the result will be logged (at debug level). There is a time cost to capturing the memory snapshots, so it is recommended to only enable this feature if you are actively inspecting the model cache's behaviour. device: Preferred execution device. `auto` will choose the device depending on the hardware platform and the installed torch capabilities.
Valid values: `auto`, `cpu`, `cuda`, `cuda:1`, `mps` - precision: Floating point precision. `float16` will consume half the memory of `float32` but produce slightly lower-quality images. The `auto` setting will guess the proper precision based on your video card and operating system.
Valid values: `auto`, `float16`, `bfloat16`, `float32`, `autocast` + precision: Floating point precision. `float16` will consume half the memory of `float32` but produce slightly lower-quality images. The `auto` setting will guess the proper precision based on your video card and operating system.
Valid values: `auto`, `float16`, `bfloat16`, `float32` sequential_guidance: Whether to calculate guidance in serial instead of in parallel, lowering memory requirements. attention_type: Attention type.
Valid values: `auto`, `normal`, `xformers`, `sliced`, `torch-sdp` attention_slice_size: Slice size, valid when attention_type=="sliced".
Valid values: `auto`, `balanced`, `max`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8` @@ -370,6 +370,9 @@ def migrate_v3_config_dict(config_dict: dict[str, Any]) -> InvokeAIAppConfig: # `max_vram_cache_size` was renamed to `vram` some time in v3, but both names were used if k == "max_vram_cache_size" and "vram" not in category_dict: parsed_config_dict["vram"] = v + # autocast was removed in v4.0.1 + if k == "precision" and v == "autocast": + parsed_config_dict["precision"] = "auto" if k == "conf_path": parsed_config_dict["legacy_models_yaml_path"] = v if k == "legacy_conf_dir": @@ -392,6 +395,28 @@ def migrate_v3_config_dict(config_dict: dict[str, Any]) -> InvokeAIAppConfig: return config +def migrate_v4_0_0_config_dict(config_dict: dict[str, Any]) -> InvokeAIAppConfig: + """Migrate v4.0.0 config dictionary to a current config object. + + Args: + config_dict: A dictionary of settings from a v4.0.0 config file. + + Returns: + An instance of `InvokeAIAppConfig` with the migrated settings. + """ + parsed_config_dict: dict[str, Any] = {} + for k, v in config_dict.items(): + # autocast was removed from precision in v4.0.1 + if k == "precision" and v == "autocast": + parsed_config_dict["precision"] = "auto" + else: + parsed_config_dict[k] = v + if k == "schema_version": + parsed_config_dict[k] = CONFIG_SCHEMA_VERSION + config = DefaultInvokeAIAppConfig.model_validate(parsed_config_dict) + return config + + def load_and_migrate_config(config_path: Path) -> InvokeAIAppConfig: """Load and migrate a config file to the latest version. @@ -418,17 +443,21 @@ def load_and_migrate_config(config_path: Path) -> InvokeAIAppConfig: raise RuntimeError(f"Failed to load and migrate v3 config file {config_path}: {e}") from e migrated_config.write_file(config_path) return migrated_config - else: - # Attempt to load as a v4 config file - try: - # Meta is not included in the model fields, so we need to validate it separately - config = InvokeAIAppConfig.model_validate(loaded_config_dict) - assert ( - config.schema_version == CONFIG_SCHEMA_VERSION - ), f"Invalid schema version, expected {CONFIG_SCHEMA_VERSION}: {config.schema_version}" - return config - except Exception as e: - raise RuntimeError(f"Failed to load config file {config_path}: {e}") from e + + if loaded_config_dict["schema_version"] == "4.0.0": + loaded_config_dict = migrate_v4_0_0_config_dict(loaded_config_dict) + loaded_config_dict.write_file(config_path) + + # Attempt to load as a v4 config file + try: + # Meta is not included in the model fields, so we need to validate it separately + config = InvokeAIAppConfig.model_validate(loaded_config_dict) + assert ( + config.schema_version == CONFIG_SCHEMA_VERSION + ), f"Invalid schema version, expected {CONFIG_SCHEMA_VERSION}: {config.schema_version}" + return config + except Exception as e: + raise RuntimeError(f"Failed to load config file {config_path}: {e}") from e @lru_cache(maxsize=1) diff --git a/invokeai/app/services/model_install/model_install_default.py b/invokeai/app/services/model_install/model_install_default.py index 20cfc1c4ff..6a3117bcb8 100644 --- a/invokeai/app/services/model_install/model_install_default.py +++ b/invokeai/app/services/model_install/model_install_default.py @@ -13,6 +13,7 @@ from shutil import copyfile, copytree, move, rmtree from tempfile import mkdtemp from typing import Any, Dict, List, Optional, Union +import torch import yaml from huggingface_hub import HfFolder from pydantic.networks import AnyHttpUrl @@ -42,7 +43,7 @@ from invokeai.backend.model_manager.metadata.metadata_base import HuggingFaceMet from invokeai.backend.model_manager.probe import ModelProbe from invokeai.backend.model_manager.search import ModelSearch from invokeai.backend.util import InvokeAILogger -from invokeai.backend.util.devices import choose_precision, choose_torch_device +from invokeai.backend.util.devices import TorchDevice from .model_install_base import ( MODEL_SOURCE_TO_TYPE_MAP, @@ -634,11 +635,10 @@ class ModelInstallService(ModelInstallServiceBase): self._next_job_id += 1 return id - @staticmethod - def _guess_variant() -> Optional[ModelRepoVariant]: + def _guess_variant(self) -> Optional[ModelRepoVariant]: """Guess the best HuggingFace variant type to download.""" - precision = choose_precision(choose_torch_device()) - return ModelRepoVariant.FP16 if precision == "float16" else None + precision = TorchDevice.choose_torch_dtype() + return ModelRepoVariant.FP16 if precision == torch.float16 else None def _import_local_model(self, source: LocalModelSource, config: Optional[Dict[str, Any]]) -> ModelInstallJob: return ModelInstallJob( @@ -754,6 +754,8 @@ class ModelInstallService(ModelInstallServiceBase): self._download_cache[download_job.source] = install_job # matches a download job to an install job install_job.download_parts.add(download_job) + # only start the jobs once install_job.download_parts is fully populated + for download_job in install_job.download_parts: self._download_queue.submit_download_job( download_job, on_start=self._download_started_callback, @@ -762,6 +764,7 @@ class ModelInstallService(ModelInstallServiceBase): on_error=self._download_error_callback, on_cancelled=self._download_cancelled_callback, ) + return install_job def _stat_size(self, path: Path) -> int: diff --git a/invokeai/app/services/model_manager/model_manager_default.py b/invokeai/app/services/model_manager/model_manager_default.py index de6e5f09d8..1a2b9a3402 100644 --- a/invokeai/app/services/model_manager/model_manager_default.py +++ b/invokeai/app/services/model_manager/model_manager_default.py @@ -1,12 +1,14 @@ # Copyright (c) 2023 Lincoln D. Stein and the InvokeAI Team """Implementation of ModelManagerServiceBase.""" +from typing import Optional + import torch from typing_extensions import Self from invokeai.app.services.invoker import Invoker from invokeai.backend.model_manager.load import ModelCache, ModelConvertCache, ModelLoaderRegistry -from invokeai.backend.util.devices import choose_torch_device +from invokeai.backend.util.devices import TorchDevice from invokeai.backend.util.logging import InvokeAILogger from ..config import InvokeAIAppConfig @@ -67,7 +69,7 @@ class ModelManagerService(ModelManagerServiceBase): model_record_service: ModelRecordServiceBase, download_queue: DownloadQueueServiceBase, events: EventServiceBase, - execution_device: torch.device = choose_torch_device(), + execution_device: Optional[torch.device] = None, ) -> Self: """ Construct the model manager service instance. @@ -82,7 +84,7 @@ class ModelManagerService(ModelManagerServiceBase): max_vram_cache_size=app_config.vram, lazy_offloading=app_config.lazy_offload, logger=logger, - execution_device=execution_device, + execution_device=execution_device or TorchDevice.choose_torch_device(), ) convert_cache = ModelConvertCache(cache_path=app_config.convert_cache_path, max_size=app_config.convert_cache) loader = ModelLoadService( diff --git a/invokeai/backend/image_util/depth_anything/__init__.py b/invokeai/backend/image_util/depth_anything/__init__.py index ccac2ba949..c854fba3f2 100644 --- a/invokeai/backend/image_util/depth_anything/__init__.py +++ b/invokeai/backend/image_util/depth_anything/__init__.py @@ -13,7 +13,7 @@ from invokeai.app.services.config.config_default import get_config from invokeai.app.util.download_with_progress import download_with_progress_bar from invokeai.backend.image_util.depth_anything.model.dpt import DPT_DINOv2 from invokeai.backend.image_util.depth_anything.utilities.util import NormalizeImage, PrepareForNet, Resize -from invokeai.backend.util.devices import choose_torch_device +from invokeai.backend.util.devices import TorchDevice from invokeai.backend.util.logging import InvokeAILogger config = get_config() @@ -56,7 +56,7 @@ class DepthAnythingDetector: def __init__(self) -> None: self.model = None self.model_size: Union[Literal["large", "base", "small"], None] = None - self.device = choose_torch_device() + self.device = TorchDevice.choose_torch_device() def load_model(self, model_size: Literal["large", "base", "small"] = "small"): DEPTH_ANYTHING_MODEL_PATH = config.models_path / DEPTH_ANYTHING_MODELS[model_size]["local"] @@ -81,7 +81,7 @@ class DepthAnythingDetector: self.model.load_state_dict(torch.load(DEPTH_ANYTHING_MODEL_PATH.as_posix(), map_location="cpu")) self.model.eval() - self.model.to(choose_torch_device()) + self.model.to(self.device) return self.model def __call__(self, image: Image.Image, resolution: int = 512) -> Image.Image: @@ -94,7 +94,7 @@ class DepthAnythingDetector: image_height, image_width = np_image.shape[:2] np_image = transform({"image": np_image})["image"] - tensor_image = torch.from_numpy(np_image).unsqueeze(0).to(choose_torch_device()) + tensor_image = torch.from_numpy(np_image).unsqueeze(0).to(self.device) with torch.no_grad(): depth = self.model(tensor_image) diff --git a/invokeai/backend/image_util/dw_openpose/wholebody.py b/invokeai/backend/image_util/dw_openpose/wholebody.py index 35d340640d..84f5afa989 100644 --- a/invokeai/backend/image_util/dw_openpose/wholebody.py +++ b/invokeai/backend/image_util/dw_openpose/wholebody.py @@ -7,7 +7,7 @@ import onnxruntime as ort from invokeai.app.services.config.config_default import get_config from invokeai.app.util.download_with_progress import download_with_progress_bar -from invokeai.backend.util.devices import choose_torch_device +from invokeai.backend.util.devices import TorchDevice from .onnxdet import inference_detector from .onnxpose import inference_pose @@ -28,9 +28,9 @@ config = get_config() class Wholebody: def __init__(self): - device = choose_torch_device() + device = TorchDevice.choose_torch_device() - providers = ["CUDAExecutionProvider"] if device == "cuda" else ["CPUExecutionProvider"] + providers = ["CUDAExecutionProvider"] if device.type == "cuda" else ["CPUExecutionProvider"] DET_MODEL_PATH = config.models_path / DWPOSE_MODELS["yolox_l.onnx"]["local"] download_with_progress_bar("yolox_l.onnx", DWPOSE_MODELS["yolox_l.onnx"]["url"], DET_MODEL_PATH) diff --git a/invokeai/backend/image_util/infill_methods/lama.py b/invokeai/backend/image_util/infill_methods/lama.py index fa354aeed1..4268ec773d 100644 --- a/invokeai/backend/image_util/infill_methods/lama.py +++ b/invokeai/backend/image_util/infill_methods/lama.py @@ -8,7 +8,7 @@ from PIL import Image import invokeai.backend.util.logging as logger from invokeai.app.services.config.config_default import get_config from invokeai.app.util.download_with_progress import download_with_progress_bar -from invokeai.backend.util.devices import choose_torch_device +from invokeai.backend.util.devices import TorchDevice def norm_img(np_img): @@ -29,7 +29,7 @@ def load_jit_model(url_or_path, device): class LaMA: def __call__(self, input_image: Image.Image, *args: Any, **kwds: Any) -> Any: - device = choose_torch_device() + device = TorchDevice.choose_torch_device() model_location = get_config().models_path / "core/misc/lama/lama.pt" if not model_location.exists(): diff --git a/invokeai/backend/image_util/realesrgan/realesrgan.py b/invokeai/backend/image_util/realesrgan/realesrgan.py index c06504b608..663a323967 100644 --- a/invokeai/backend/image_util/realesrgan/realesrgan.py +++ b/invokeai/backend/image_util/realesrgan/realesrgan.py @@ -11,7 +11,7 @@ from cv2.typing import MatLike from tqdm import tqdm from invokeai.backend.image_util.basicsr.rrdbnet_arch import RRDBNet -from invokeai.backend.util.devices import choose_torch_device +from invokeai.backend.util.devices import TorchDevice """ Adapted from https://github.com/xinntao/Real-ESRGAN/blob/master/realesrgan/utils.py @@ -65,7 +65,7 @@ class RealESRGAN: self.pre_pad = pre_pad self.mod_scale: Optional[int] = None self.half = half - self.device = choose_torch_device() + self.device = TorchDevice.choose_torch_device() loadnet = torch.load(model_path, map_location=torch.device("cpu")) diff --git a/invokeai/backend/image_util/safety_checker.py b/invokeai/backend/image_util/safety_checker.py index 7bceae8da7..60dcd93fcc 100644 --- a/invokeai/backend/image_util/safety_checker.py +++ b/invokeai/backend/image_util/safety_checker.py @@ -13,7 +13,7 @@ from transformers import AutoFeatureExtractor import invokeai.backend.util.logging as logger from invokeai.app.services.config.config_default import get_config -from invokeai.backend.util.devices import choose_torch_device +from invokeai.backend.util.devices import TorchDevice from invokeai.backend.util.silence_warnings import SilenceWarnings CHECKER_PATH = "core/convert/stable-diffusion-safety-checker" @@ -51,7 +51,7 @@ class SafetyChecker: cls._load_safety_checker() if cls.safety_checker is None or cls.feature_extractor is None: return False - device = choose_torch_device() + device = TorchDevice.choose_torch_device() features = cls.feature_extractor([image], return_tensors="pt") features.to(device) cls.safety_checker.to(device) diff --git a/invokeai/backend/model_manager/load/load_default.py b/invokeai/backend/model_manager/load/load_default.py index 6774fc2989..a58741763f 100644 --- a/invokeai/backend/model_manager/load/load_default.py +++ b/invokeai/backend/model_manager/load/load_default.py @@ -18,7 +18,7 @@ from invokeai.backend.model_manager.load.load_base import LoadedModel, ModelLoad from invokeai.backend.model_manager.load.model_cache.model_cache_base import ModelCacheBase, ModelLockerBase from invokeai.backend.model_manager.load.model_util import calc_model_size_by_data, calc_model_size_by_fs from invokeai.backend.model_manager.load.optimizations import skip_torch_weight_init -from invokeai.backend.util.devices import choose_torch_device, torch_dtype +from invokeai.backend.util.devices import TorchDevice # TO DO: The loader is not thread safe! @@ -37,7 +37,7 @@ class ModelLoader(ModelLoaderBase): self._logger = logger self._ram_cache = ram_cache self._convert_cache = convert_cache - self._torch_dtype = torch_dtype(choose_torch_device()) + self._torch_dtype = TorchDevice.choose_torch_dtype() def load_model(self, model_config: AnyModelConfig, submodel_type: Optional[SubModelType] = None) -> LoadedModel: """ diff --git a/invokeai/backend/model_manager/load/model_cache/model_cache_default.py b/invokeai/backend/model_manager/load/model_cache/model_cache_default.py index 2ba52d466c..2ffe954e11 100644 --- a/invokeai/backend/model_manager/load/model_cache/model_cache_default.py +++ b/invokeai/backend/model_manager/load/model_cache/model_cache_default.py @@ -30,15 +30,12 @@ import torch from invokeai.backend.model_manager import AnyModel, SubModelType from invokeai.backend.model_manager.load.memory_snapshot import MemorySnapshot, get_pretty_snapshot_diff -from invokeai.backend.util.devices import choose_torch_device +from invokeai.backend.util.devices import TorchDevice from invokeai.backend.util.logging import InvokeAILogger from .model_cache_base import CacheRecord, CacheStats, ModelCacheBase, ModelLockerBase from .model_locker import ModelLocker -if choose_torch_device() == torch.device("mps"): - from torch import mps - # Maximum size of the cache, in gigs # Default is roughly enough to hold three fp16 diffusers models in RAM simultaneously DEFAULT_MAX_CACHE_SIZE = 6.0 @@ -244,9 +241,7 @@ class ModelCache(ModelCacheBase[AnyModel]): f"Removing {cache_entry.key} from VRAM to free {(cache_entry.size/GIG):.2f}GB; vram free = {(torch.cuda.memory_allocated()/GIG):.2f}GB" ) - torch.cuda.empty_cache() - if choose_torch_device() == torch.device("mps"): - mps.empty_cache() + TorchDevice.empty_cache() def move_model_to_device(self, cache_entry: CacheRecord[AnyModel], target_device: torch.device) -> None: """Move model into the indicated device. @@ -416,10 +411,7 @@ class ModelCache(ModelCacheBase[AnyModel]): self.stats.cleared = models_cleared gc.collect() - torch.cuda.empty_cache() - if choose_torch_device() == torch.device("mps"): - mps.empty_cache() - + TorchDevice.empty_cache() self.logger.debug(f"After making room: cached_models={len(self._cached_models)}") def _delete_cache_entry(self, cache_entry: CacheRecord[AnyModel]) -> None: diff --git a/invokeai/backend/model_manager/merge.py b/invokeai/backend/model_manager/merge.py index eb6fd45e1a..125e99be93 100644 --- a/invokeai/backend/model_manager/merge.py +++ b/invokeai/backend/model_manager/merge.py @@ -17,7 +17,7 @@ from diffusers.utils import logging as dlogging from invokeai.app.services.model_install import ModelInstallServiceBase from invokeai.app.services.model_records.model_records_base import ModelRecordChanges -from invokeai.backend.util.devices import choose_torch_device, torch_dtype +from invokeai.backend.util.devices import TorchDevice from . import ( AnyModelConfig, @@ -43,6 +43,7 @@ class ModelMerger(object): Initialize a ModelMerger object with the model installer. """ self._installer = installer + self._dtype = TorchDevice.choose_torch_dtype() def merge_diffusion_models( self, @@ -68,7 +69,7 @@ class ModelMerger(object): warnings.simplefilter("ignore") verbosity = dlogging.get_verbosity() dlogging.set_verbosity_error() - dtype = torch.float16 if variant == "fp16" else torch_dtype(choose_torch_device()) + dtype = torch.float16 if variant == "fp16" else self._dtype # Note that checkpoint_merger will not work with downloaded HuggingFace fp16 models # until upstream https://github.com/huggingface/diffusers/pull/6670 is merged and released. @@ -151,7 +152,7 @@ class ModelMerger(object): dump_path.mkdir(parents=True, exist_ok=True) dump_path = dump_path / merged_model_name - dtype = torch.float16 if variant == "fp16" else torch_dtype(choose_torch_device()) + dtype = torch.float16 if variant == "fp16" else self._dtype merged_pipe.save_pretrained(dump_path.as_posix(), safe_serialization=True, torch_dtype=dtype, variant=variant) # register model and get its unique key diff --git a/invokeai/backend/stable_diffusion/diffusers_pipeline.py b/invokeai/backend/stable_diffusion/diffusers_pipeline.py index befda72751..8b90c815ae 100644 --- a/invokeai/backend/stable_diffusion/diffusers_pipeline.py +++ b/invokeai/backend/stable_diffusion/diffusers_pipeline.py @@ -25,7 +25,7 @@ from invokeai.backend.stable_diffusion.diffusion.conditioning_data import IPAdap from invokeai.backend.stable_diffusion.diffusion.shared_invokeai_diffusion import InvokeAIDiffuserComponent from invokeai.backend.stable_diffusion.diffusion.unet_attention_patcher import UNetAttentionPatcher, UNetIPAdapterData from invokeai.backend.util.attention import auto_detect_slice_size -from invokeai.backend.util.devices import normalize_device +from invokeai.backend.util.devices import TorchDevice @dataclass @@ -255,7 +255,7 @@ class StableDiffusionGeneratorPipeline(StableDiffusionPipeline): if self.unet.device.type == "cpu" or self.unet.device.type == "mps": mem_free = psutil.virtual_memory().free elif self.unet.device.type == "cuda": - mem_free, _ = torch.cuda.mem_get_info(normalize_device(self.unet.device)) + mem_free, _ = torch.cuda.mem_get_info(TorchDevice.normalize(self.unet.device)) else: raise ValueError(f"unrecognized device {self.unet.device}") # input tensor of [1, 4, h/8, w/8] diff --git a/invokeai/backend/util/__init__.py b/invokeai/backend/util/__init__.py index 2c9cceff2c..1e4d467cd0 100644 --- a/invokeai/backend/util/__init__.py +++ b/invokeai/backend/util/__init__.py @@ -2,7 +2,6 @@ Initialization file for invokeai.backend.util """ -from .devices import choose_precision, choose_torch_device from .logging import InvokeAILogger from .util import GIG, Chdir, directory_size @@ -11,6 +10,4 @@ __all__ = [ "directory_size", "Chdir", "InvokeAILogger", - "choose_precision", - "choose_torch_device", ] diff --git a/invokeai/backend/util/devices.py b/invokeai/backend/util/devices.py index cb6b93eaac..e8380dc8bc 100644 --- a/invokeai/backend/util/devices.py +++ b/invokeai/backend/util/devices.py @@ -1,89 +1,110 @@ -from __future__ import annotations - -from contextlib import nullcontext -from typing import Literal, Optional, Union +from typing import Dict, Literal, Optional, Union import torch -from torch import autocast +from deprecated import deprecated -from invokeai.app.services.config.config_default import PRECISION, get_config +from invokeai.app.services.config.config_default import get_config +# legacy APIs +TorchPrecisionNames = Literal["float32", "float16", "bfloat16"] CPU_DEVICE = torch.device("cpu") CUDA_DEVICE = torch.device("cuda") MPS_DEVICE = torch.device("mps") +@deprecated("Use TorchDevice.choose_torch_dtype() instead.") # type: ignore +def choose_precision(device: torch.device) -> TorchPrecisionNames: + """Return the string representation of the recommended torch device.""" + torch_dtype = TorchDevice.choose_torch_dtype(device) + return PRECISION_TO_NAME[torch_dtype] + + +@deprecated("Use TorchDevice.choose_torch_device() instead.") # type: ignore def choose_torch_device() -> torch.device: - """Convenience routine for guessing which GPU device to run model on""" - config = get_config() - if config.device == "auto": - if torch.cuda.is_available(): - return torch.device("cuda") - if hasattr(torch.backends, "mps") and torch.backends.mps.is_available(): - return torch.device("mps") + """Return the torch.device to use for accelerated inference.""" + return TorchDevice.choose_torch_device() + + +@deprecated("Use TorchDevice.choose_torch_dtype() instead.") # type: ignore +def torch_dtype(device: torch.device) -> torch.dtype: + """Return the torch precision for the recommended torch device.""" + return TorchDevice.choose_torch_dtype(device) + + +NAME_TO_PRECISION: Dict[TorchPrecisionNames, torch.dtype] = { + "float32": torch.float32, + "float16": torch.float16, + "bfloat16": torch.bfloat16, +} +PRECISION_TO_NAME: Dict[torch.dtype, TorchPrecisionNames] = {v: k for k, v in NAME_TO_PRECISION.items()} + + +class TorchDevice: + """Abstraction layer for torch devices.""" + + @classmethod + def choose_torch_device(cls) -> torch.device: + """Return the torch.device to use for accelerated inference.""" + app_config = get_config() + if app_config.device != "auto": + device = torch.device(app_config.device) + elif torch.cuda.is_available(): + device = CUDA_DEVICE + elif torch.backends.mps.is_available(): + device = MPS_DEVICE else: - return CPU_DEVICE - else: - return torch.device(config.device) + device = CPU_DEVICE + return cls.normalize(device) + @classmethod + def choose_torch_dtype(cls, device: Optional[torch.device] = None) -> torch.dtype: + """Return the precision to use for accelerated inference.""" + device = device or cls.choose_torch_device() + config = get_config() + if device.type == "cuda" and torch.cuda.is_available(): + device_name = torch.cuda.get_device_name(device) + if "GeForce GTX 1660" in device_name or "GeForce GTX 1650" in device_name: + # These GPUs have limited support for float16 + return cls._to_dtype("float32") + elif config.precision == "auto": + # Default to float16 for CUDA devices + return cls._to_dtype("float16") + else: + # Use the user-defined precision + return cls._to_dtype(config.precision) -def get_torch_device_name() -> str: - device = choose_torch_device() - return torch.cuda.get_device_name(device) if device.type == "cuda" else device.type.upper() + elif device.type == "mps" and torch.backends.mps.is_available(): + if config.precision == "auto": + # Default to float16 for MPS devices + return cls._to_dtype("float16") + else: + # Use the user-defined precision + return cls._to_dtype(config.precision) + # CPU / safe fallback + return cls._to_dtype("float32") + @classmethod + def get_torch_device_name(cls) -> str: + """Return the device name for the current torch device.""" + device = cls.choose_torch_device() + return torch.cuda.get_device_name(device) if device.type == "cuda" else device.type.upper() -def choose_precision(device: torch.device) -> Literal["float32", "float16", "bfloat16"]: - """Return an appropriate precision for the given torch device.""" - app_config = get_config() - if device.type == "cuda": - device_name = torch.cuda.get_device_name(device) - if "GeForce GTX 1660" in device_name or "GeForce GTX 1650" in device_name: - # These GPUs have limited support for float16 - return "float32" - elif app_config.precision == "auto" or app_config.precision == "autocast": - # Default to float16 for CUDA devices - return "float16" - else: - # Use the user-defined precision - return app_config.precision - elif device.type == "mps": - if app_config.precision == "auto" or app_config.precision == "autocast": - # Default to float16 for MPS devices - return "float16" - else: - # Use the user-defined precision - return app_config.precision - # CPU / safe fallback - return "float32" - - -def torch_dtype(device: Optional[torch.device] = None) -> torch.dtype: - device = device or choose_torch_device() - precision = choose_precision(device) - if precision == "float16": - return torch.float16 - if precision == "bfloat16": - return torch.bfloat16 - else: - # "auto", "autocast", "float32" - return torch.float32 - - -def choose_autocast(precision: PRECISION): - """Returns an autocast context or nullcontext for the given precision string""" - # float16 currently requires autocast to avoid errors like: - # 'expected scalar type Half but found Float' - if precision == "autocast" or precision == "float16": - return autocast - return nullcontext - - -def normalize_device(device: Union[str, torch.device]) -> torch.device: - """Ensure device has a device index defined, if appropriate.""" - device = torch.device(device) - if device.index is None: - # cuda might be the only torch backend that currently uses the device index? - # I don't see anything like `current_device` for cpu or mps. - if device.type == "cuda": + @classmethod + def normalize(cls, device: Union[str, torch.device]) -> torch.device: + """Add the device index to CUDA devices.""" + device = torch.device(device) + if device.index is None and device.type == "cuda" and torch.cuda.is_available(): device = torch.device(device.type, torch.cuda.current_device()) - return device + return device + + @classmethod + def empty_cache(cls) -> None: + """Clear the GPU device cache.""" + if torch.backends.mps.is_available(): + torch.mps.empty_cache() + if torch.cuda.is_available(): + torch.cuda.empty_cache() + + @classmethod + def _to_dtype(cls, precision_name: TorchPrecisionNames) -> torch.dtype: + return NAME_TO_PRECISION[precision_name] diff --git a/invokeai/frontend/web/index.html b/invokeai/frontend/web/index.html index 07db8cd3ff..d74db800da 100644 --- a/invokeai/frontend/web/index.html +++ b/invokeai/frontend/web/index.html @@ -8,7 +8,7 @@ Invoke - Community Edition - +