mirror of https://github.com/invoke-ai/InvokeAI synced 2024-08-30 20:32:17 +00:00

History

Lincoln Stein a3cb5da130 Improve RAM<->VRAM memory copy performance in LoRA patching and elsewhere (#6490 ) * allow model patcher to optimize away the unpatching step when feasible * remove lazy_offloading functionality * allow model patcher to optimize away the unpatching step when feasible * remove lazy_offloading functionality * do not save original weights if there is a CPU copy of state dict * Update invokeai/backend/model_manager/load/load_base.py Co-authored-by: Ryan Dick <ryanjdick3@gmail.com> * documentation fixes requested during penultimate review * add non-blocking=True parameters to several torch.nn.Module.to() calls, for slight performance increases * fix ruff errors * prevent crash on non-cuda-enabled systems --------- Co-authored-by: Lincoln Stein <lstein@gmail.com> Co-authored-by: Kent Keirsey <31807370+hipsterusername@users.noreply.github.com> Co-authored-by: Ryan Dick <ryanjdick3@gmail.com>		2024-06-13 17:10:03 +00:00
..
__init__.py	chore: flake8 cleanup	2023-09-05 12:07:12 +12:00
ip_adapter.py	Improve RAM<->VRAM memory copy performance in LoRA patching and elsewhere (#6490 )	2024-06-13 17:10:03 +00:00
ip_attention_weights.py	Fix handling of scales with multiple IP-Adapters.	2023-10-06 20:43:43 -04:00
README.md	(minor) Update documentation to reflect that a bug was fixed in InvokeAI/ip_adapter_sdxl_vit_h by `e178288fb6`	2023-09-20 20:18:33 -04:00
resampler.py	wip: Initial implementation of safetensor support for IP Adapter	2024-04-03 12:39:52 +05:30

README.md

IP-Adapter Model Formats

The official IP-Adapter models are released here: h94/IP-Adapter

This official model repo does not integrate well with InvokeAI's current approach to model management, so we have defined a new file structure for IP-Adapter models. The InvokeAI format is described below.

CLIP Vision Models

CLIP Vision models are organized in `diffusers`` format. The expected directory structure is:

ip_adapter_sd_image_encoder/
├── config.json
└── model.safetensors

IP-Adapter Models

IP-Adapter models are stored in a directory containing two files

image_encoder.txt: A text file containing the model identifier for the CLIP Vision encoder that is intended to be used with this IP-Adapter model.
ip_adapter.bin: The IP-Adapter weights.

Sample directory structure:

ip_adapter_sd15/
├── image_encoder.txt
└── ip_adapter.bin

Why save the weights in a .safetensors file?

The weights in ip_adapter.bin are stored in a nested dict, which is not supported by safetensors. This could be solved by splitting ip_adapter.bin into multiple files, but for now we have decided to maintain consistency with the checkpoint structure used in the official h94/IP-Adapter repo.

InvokeAI Hosted IP-Adapters

Image Encoders:

IP-Adapters: