Commit Graph

12392 Commits

Author SHA1 Message Date
Ryan Dick
fb5db32bb0 Make it run (with artifacts). 2024-08-19 22:12:12 +00:00
Ryan Dick
823c663e1b WIP on moving from diffusers to FLUX 2024-08-16 20:22:49 +00:00
Ryan Dick
d40c9ff60a More improvements for LLM.int8() - not fully tested. 2024-08-15 19:59:31 +00:00
Ryan Dick
373b46867a LLM.int8() quantization is working, but still some rough edges to solve. 2024-08-15 19:34:34 +00:00
Ryan Dick
dc66952491 Clean up NF4 implementation. 2024-08-15 16:30:47 +00:00
Ryan Dick
1b80832b22 NF4 inference working 2024-08-14 23:30:53 +00:00
Ryan Dick
96b0450b20 NF4 loading working... I think. 2024-08-14 14:47:03 +00:00
Ryan Dick
45792cc152 wip 2024-08-14 04:06:16 +00:00
Ryan Dick
f0baf880b5 Split a FluxTextEncoderInvocation out from the FluxTextToImageInvocation. This has the advantage that we benfit from automatic caching when the prompt isn't changed. 2024-08-12 18:23:02 +00:00
Ryan Dick
a8a2fc106d Make quantized loading fast for both T5XXL and FLUX transformer. 2024-08-09 19:54:09 +00:00
Ryan Dick
d23ad1818d Make quantized loading fast. 2024-08-09 16:39:43 +00:00
Ryan Dick
4181ab654b WIP - experimentation 2024-08-09 16:23:37 +00:00
Ryan Dick
1c97360f9f Make float16 inference work with FLUX on 24GB GPU. 2024-08-08 18:12:04 -04:00
Ryan Dick
74d6fceeb6 Add support for 8-bit quantizatino of the FLUX T5XXL text encoder. 2024-08-08 18:23:20 +00:00
Ryan Dick
766ddc18dc Make 8-bit quantization save/reload work for the FLUX transformer. Reload is still very slow with the current optimum.quanto implementation. 2024-08-08 16:40:11 +00:00
Ryan Dick
e6ff7488a1 Minor improvements to FLUX workflow. 2024-08-07 22:10:09 +00:00
Ryan Dick
89a652cfcd Got FLUX schnell working with 8-bit quantization. Still lots of rough edges to clean up. 2024-08-07 19:50:03 +00:00
Ryan Dick
b227b9059d Use the FluxPipeline.encode_prompt() api rather than trying to run the two text encoders separately. 2024-08-07 15:12:01 +00:00
Ryan Dick
3599a4a3e4 Add sentencepiece dependency for the T5 tokenizer. 2024-08-07 14:18:19 +00:00
Ryan Dick
5dd619e137 First draft of FluxTextToImageInvocation. 2024-08-06 21:51:22 +00:00
Ryan Dick
7d447cbb88 Update HF download logic to work for black-forest-labs/FLUX.1-schnell. 2024-08-06 19:34:49 +00:00
Ryan Dick
3bbba7e4b1 Update imports for compatibility with bumped diffusers version. 2024-08-06 17:56:36 +00:00
Ryan Dick
b1845019fe Bump diffusers version to include FLUX support. 2024-08-06 11:52:05 -04:00
Hosted Weblate
140670d00e translationBot(ui): update translation files
Updated by "Cleanup translation files" hook in Weblate.

translationBot(ui): update translation files

Updated by "Cleanup translation files" hook in Weblate.

Co-authored-by: Hosted Weblate <hosted@weblate.org>
Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/
Translation: InvokeAI/Web UI
2024-08-06 17:54:47 +10:00
Phrixus2023
70233fae5d translationBot(ui): update translation (Chinese (Simplified))
Currently translated at 98.1% (1296 of 1321 strings)

Co-authored-by: Phrixus2023 <920414016@qq.com>
Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/zh_Hans/
Translation: InvokeAI/Web UI
2024-08-06 17:54:47 +10:00
Alexander Eichhorn
6f457a6c4c translationBot(ui): update translation (German)
Currently translated at 65.1% (860 of 1321 strings)

Co-authored-by: Alexander Eichhorn <pfannkuchensack@einfach-doof.de>
Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/de/
Translation: InvokeAI/Web UI
2024-08-06 17:54:47 +10:00
B N
5c319f5356 translationBot(ui): update translation (German)
Currently translated at 64.8% (857 of 1321 strings)

Co-authored-by: B N <berndnieschalk@gmail.com>
Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/de/
Translation: InvokeAI/Web UI
2024-08-06 17:54:47 +10:00
Riccardo Giovanetti
991a04f090 translationBot(ui): update translation (Italian)
Currently translated at 98.6% (1303 of 1321 strings)

translationBot(ui): update translation (Italian)

Currently translated at 98.6% (1302 of 1320 strings)

translationBot(ui): update translation (Italian)

Currently translated at 98.6% (1294 of 1312 strings)

Co-authored-by: Riccardo Giovanetti <riccardo.giovanetti@gmail.com>
Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/it/
Translation: InvokeAI/Web UI
2024-08-06 17:54:47 +10:00
psychedelicious
c39fa75113 docs(ui): add comment in useIsTooLargeToUpscale 2024-08-06 11:49:35 +10:00
psychedelicious
f7863e17ce docs(ui): add docstring for maxUpscaleDimension 2024-08-06 11:49:35 +10:00
psychedelicious
7c526390ed fix(ui): compare upscaledPixels vs square of max dimension 2024-08-06 11:49:35 +10:00
Mary Hipp
2cff20f87a update translations, change config value to be dimension instead of total pixels 2024-08-06 11:49:35 +10:00
Mary Hipp
90ec757802 lint 2024-08-06 11:49:35 +10:00
Mary Hipp
4b85dfcefe (ui): restore optioanl limit on upcsale output resolution 2024-08-06 11:49:35 +10:00
Mary Hipp
21deefdc41 (ui): add image resolution badge to initial upscale image 2024-08-06 11:49:35 +10:00
psychedelicious
4d4f921a4e build: exclude matplotlib 3.9.1
There was a problem w/ this release on windows and the builds were pulled from pypi. When installing invoke on windows, pip attempts to build from source, but most (all?) systems won't have the prerequisites for this and installs fail.

This also affects GH actions.

The simple fix is to exclude version 3.9.1 from our deps.

For more information, see https://github.com/matplotlib/matplotlib/issues/28551
2024-08-05 08:38:44 +10:00
psychedelicious
98db8f395b feat(app): clean up DiskImageStorage types 2024-08-04 09:43:20 +10:00
psychedelicious
f465a956a3 feat(ui): remove "images can be restored" messages 2024-08-04 09:43:20 +10:00
psychedelicious
9edb02d7ef build: remove send2trash dependency 2024-08-04 09:43:20 +10:00
psychedelicious
6c4cf58a31 feat(app): delete model_images instead of using send2trash 2024-08-04 09:43:20 +10:00
psychedelicious
08993c0d29 feat(app): delete images instead of using send2trash
Closes #6709
2024-08-04 09:43:20 +10:00
Mary Hipp
571ba87e13 fix(ui): include upscale metadata for SDXL multidiffusion 2024-08-01 21:30:42 -04:00
Ryan Dick
f27b6e2b44
Add Grounded SAM support (text prompt image segmentation) (#6701)
## Summary

This PR enables Grounded SAM workflows
(https://arxiv.org/pdf/2401.14159) via the following:
- `GroundingDinoInvocation` for running a Grounding DINO model.
- `SegmentAnythingModelInvocation` for running a SAM model.
- `MaskTensorToImageInvocation` for convenient visualization.

Other notes:
- Uses the transformers implementation of Grounding DINO and SAM.
- The new models are treated as 'utility models' meaning that they are
not visible in the Models tab, and are downloaded automatically the
first time that they are used.

<img width="874" alt="image"
src="https://github.com/user-attachments/assets/1cbaa97d-0e27-4943-86b1-dc7327ba8675">

## Example

Input image

![be10ec0c-20a8-4ac7-840e-d1a05fffdb6a](https://github.com/user-attachments/assets/bf21572c-635d-4703-b4ab-7aba658a9671)

Prompt: "wheels", all other configs default
Result:

![2221c44e-64e6-4b18-b4cb-610514b7a554](https://github.com/user-attachments/assets/344b91f4-7f4a-4b70-8e2e-3b4a0e55176d)

## Related Issues / Discussions

Thanks to @blessedcoolant for the initial draft here:
https://github.com/invoke-ai/InvokeAI/pull/6678

## QA Instructions

Manual tests:
- [ ] Test that default settings work well.
- [ ] Test with / without apply_polygon_refinement
- [ ] Test mask_filter options
- [ ] Test detection_threshold values
- [ ] Test RGB input image
- [ ] Test RGBA input image
- [ ] Test grayscale input image
- [ ] Smoke test that an empty mask is returned when 0 objects are
detected
- [ ] Test on CPU
- [ ] Test on MPS (Works on Mac OS, but had to force both models to run
on CPU instead of MPS)

Performance:
- Peak GPU memory utilization with both Grounding DINO and SAM models
loaded is ~4.5GB. (The models do not need to be loaded at the same time,
so could be offloaded by the MM if needed.)
- On an RTX4090, with the models already cached, node execution takes
~0.6 secs.
- On my CPU, with the models cached, node execution takes ~10secs.

## Merge Plan

No special instructions.

## Checklist

- [x] _The PR has a short but descriptive title, suitable for a
changelog_
- [ ] _Tests added / updated (if applicable)_
- [x] _Documentation added / updated (if applicable)_
2024-08-01 20:40:18 +02:00
Ryan Dick
981475a624
Merge branch 'main' into ryan/grounded-sam 2024-08-01 20:30:35 +02:00
Ryan Dick
27ac61a4fb Expose all model options in the GroundingDinoInvocation and the SegmentAnythingInvocation. 2024-08-01 14:23:32 -04:00
Ryan Dick
675ffc2757 Remove BoundingBoxInvocation field name overrides. 2024-08-01 14:05:44 -04:00
Ryan Dick
44b21f10f1 Add a pydantic model_validator to BoundingBoxField to check the validity of the coords. 2024-08-01 14:00:57 -04:00
Ryan Dick
c6d49e8b1f Shorten SegmentAnythingInvocation and GroundingDinoInvocatino docstrings, since they are used as the invocation descriptions in the UI. 2024-08-01 10:17:42 -04:00
Ryan Dick
e6a512aa86 (minor) Tweak order of mask operations. 2024-08-01 10:12:24 -04:00
Ryan Dick
c3a6a6fb22 Rename SegmentAnythingModelInvocation -> SegmentAnythingInvocation. 2024-08-01 10:00:36 -04:00