38343917f8
In #6490 we enabled non-blocking torch device transfers throughout the model manager's memory management code. When using this torch feature, torch attempts to wait until the tensor transfer has completed before allowing any access to the tensor. Theoretically, that should make this a safe feature to use. This provides a small performance improvement but causes race conditions in some situations. Specific platforms/systems are affected, and complicated data dependencies can make this unsafe. - Intermittent black images on MPS devices - reported on discord and #6545, fixed with special handling in #6549. - Intermittent OOMs and black images on a P4000 GPU on Windows - reported in #6613, fixed in this commit. On my system, I haven't experience any issues with generation, but targeted testing of non-blocking ops did expose a race condition when moving tensors from CUDA to CPU. One workaround is to use torch streams with manual sync points. Our application logic is complicated enough that this would be a lot of work and feels ripe for edge cases and missed spots. Much safer is to fully revert non-locking - which is what this change does. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| ip_adapter.py | ||
| ip_attention_weights.py | ||
| README.md | ||
| resampler.py | ||
IP-Adapter Model Formats
The official IP-Adapter models are released here: h94/IP-Adapter
This official model repo does not integrate well with InvokeAI's current approach to model management, so we have defined a new file structure for IP-Adapter models. The InvokeAI format is described below.
CLIP Vision Models
CLIP Vision models are organized in `diffusers`` format. The expected directory structure is:
ip_adapter_sd_image_encoder/
├── config.json
└── model.safetensors
IP-Adapter Models
IP-Adapter models are stored in a directory containing two files
image_encoder.txt: A text file containing the model identifier for the CLIP Vision encoder that is intended to be used with this IP-Adapter model.ip_adapter.bin: The IP-Adapter weights.
Sample directory structure:
ip_adapter_sd15/
├── image_encoder.txt
└── ip_adapter.bin
Why save the weights in a .safetensors file?
The weights in ip_adapter.bin are stored in a nested dict, which is not supported by safetensors. This could be solved by splitting ip_adapter.bin into multiple files, but for now we have decided to maintain consistency with the checkpoint structure used in the official h94/IP-Adapter repo.
InvokeAI Hosted IP-Adapters
Image Encoders:
IP-Adapters: