* add basic functionality for model metadata fetching from hf and civitai
* add storage
* start unit tests
* add unit tests and documentation
* add missing dependency for pytests
* remove redundant fetch; add modified/published dates; updated docs
* add code to select diffusers files based on the variant type
* implement Civitai installs
* make huggingface parallel downloading work
* add unit tests for model installation manager
- Fixed race condition on selection of download destination path
- Add fixtures common to several model_manager_2 unit tests
- Added dummy model files for testing diffusers and safetensors downloading/probing
- Refactored code for selecting proper variant from list of huggingface repo files
- Regrouped ordering of methods in model_install_default.py
* improve Civitai model downloading
- Provide a better error message when Civitai requires an access token (doesn't give a 403 forbidden, but redirects
to the HTML of an authorization page -- arrgh)
- Handle case of Civitai providing a primary download link plus additional links for VAEs, config files, etc
* add routes for retrieving metadata and tags
* code tidying and documentation
* fix ruff errors
* add file needed to maintain test root diretory in repo for unit tests
* fix self->cls in classmethod
* add pydantic plugin for mypy
* use TestSession instead of requests.Session to prevent any internet activity
improve logging
fix error message formatting
fix logging again
fix forward vs reverse slash issue in Windows install tests
* Several fixes of problems detected during PR review:
- Implement cancel_model_install_job and get_model_install_job routes
to allow for better control of model download and install.
- Fix thread deadlock that occurred after cancelling an install.
- Remove unneeded pytest_plugins section from tests/conftest.py
- Remove unused _in_terminal_state() from model_install_default.
- Remove outdated documentation from several spots.
- Add workaround for Civitai API results which don't return correct
URL for the default model.
* fix docs and tests to match get_job_by_source() rather than get_job()
* Update invokeai/backend/model_manager/metadata/fetch/huggingface.py
Co-authored-by: Ryan Dick <ryanjdick3@gmail.com>
* Call CivitaiMetadata.model_validate_json() directly
Co-authored-by: Ryan Dick <ryanjdick3@gmail.com>
* Second round of revisions suggested by @ryanjdick:
- Fix type mismatch in `list_all_metadata()` route.
- Do not have a default value for the model install job id
- Remove static class variable declarations from non Pydantic classes
- Change `id` field to `model_id` for the sqlite3 `model_tags` table.
- Changed AFTER DELETE triggers to ON DELETE CASCADE for the metadata and tags tables.
- Made the `id` field of the `model_metadata` table into a primary key to achieve uniqueness.
* Code cleanup suggested in PR review:
- Narrowed the declaration of the `parts` attribute of the download progress event
- Removed auto-conversion of str to Url in Url-containing sources
- Fixed handling of `InvalidModelConfigException`
- Made unknown sources raise `NotImplementedError` rather than `Exception`
- Improved status reporting on cached HuggingFace access tokens
* Multiple fixes:
- `job.total_size` returns a valid size for locally installed models
- new route `list_models` returns a paged summary of model, name,
description, tags and other essential info
- fix a few type errors
* consolidated all invokeai root pytest fixtures into a single location
* Update invokeai/backend/model_manager/metadata/metadata_store.py
Co-authored-by: psychedelicious <4822129+psychedelicious@users.noreply.github.com>
* Small tweaks in response to review comments:
- Remove flake8 configuration from pyproject.toml
- Use `id` rather than `modelId` for huggingface `ModelInfo` object
- Use `last_modified` rather than `LastModified` for huggingface `ModelInfo` object
- Add `sha256` field to file metadata downloaded from huggingface
- Add `Invoker` argument to the model installer `start()` and `stop()` routines
(but made it optional in order to facilitate use of the service outside the API)
- Removed redundant `PRAGMA foreign_keys` from metadata store initialization code.
* Additional tweaks and minor bug fixes
- Fix calculation of aggregate diffusers model size to only count the
size of files, not files + directories (which gives different unit test
results on different filesystems).
- Refactor _get_metadata() and _get_download_urls() to have distinct code paths
for Civitai, HuggingFace and URL sources.
- Forward the `inplace` flag from the source to the job and added unit test for this.
- Attach cached model metadata to the job rather than to the model install service.
* fix unit test that was breaking on windows due to CR/LF changing size of test json files
* fix ruff formatting
* a few last minor fixes before merging:
- Turn job `error` and `error_type` into properties derived from the exception.
- Add TODO comment about the reason for handling temporary directory destruction
manually rather than using tempfile.tmpdir().
* add unit tests for reporting HTTP download errors
---------
Co-authored-by: Lincoln Stein <lstein@gmail.com>
Co-authored-by: Ryan Dick <ryanjdick3@gmail.com>
Co-authored-by: psychedelicious <4822129+psychedelicious@users.noreply.github.com>
* add base definition of download manager
* basic functionality working
* add unit tests for download queue
* add documentation and FastAPI route
* fix docs
* add missing test dependency; fix import ordering
* fix file path length checking on windows
* fix ruff check error
* move release() into the __del__ method
* disable testing of stderr messages due to issues with pytest capsys fixture
* fix unsorted imports
* harmonized implementation of start() and stop() calls in download and & install modules
* Update invokeai/app/services/download/download_base.py
Co-authored-by: Ryan Dick <ryanjdick3@gmail.com>
* replace test datadir fixture with tmp_path
* replace DownloadJobBase->DownloadJob in download manager documentation
* make source and dest arguments to download_queue.download() an AnyHttpURL and Path respectively
* fix pydantic typecheck errors in the download unit test
* ruff formatting
* add "job cancelled" as an event rather than an exception
* fix ruff errors
* Update invokeai/app/services/download/download_default.py
Co-authored-by: psychedelicious <4822129+psychedelicious@users.noreply.github.com>
* use threading.Event to stop service worker threads; handle unfinished job edge cases
* remove dangling STOP job definition
* fix ruff complaint
* fix ruff check again
* avoid race condition when start() and stop() are called simultaneously from different threads
* avoid race condition in stop() when a job becomes active while shutting down
---------
Co-authored-by: Lincoln Stein <lstein@gmail.com>
Co-authored-by: Ryan Dick <ryanjdick3@gmail.com>
Co-authored-by: psychedelicious <4822129+psychedelicious@users.noreply.github.com>
Co-authored-by: Kent Keirsey <31807370+hipsterusername@users.noreply.github.com>
* node-FaceTools
* Added more documentation for facetools
* invert FaceMask masking
- FaceMask had face protected and surroundings change by default (face white, else black)
- Change to how FaceOff/others work: the opposite where surroundings protected, face changes by default (face black, else white)
* reflect changed facemask behaviour in docs
* add FaceOff+FaceMask workflows
- Add FaceOff and FaceMask example workflows to docs/workflows
* add FaceMask+FaceOff workflows to exampleworkflows.md
- used invokeai URL paths mimicking other workflow URLs, hopefully they translate when/if merged
* inheriting, typehints, black/isort/flake8
- modified FaceMask and FaceOff output classes to inherit base image, height, width from ImageOutput
- Added type annotations to helper functions, required some reworking of code's stored data
* remove credit header
- Was in my personal/repo copy, don't think it's necessary if merged.
* Optionals & image declaration duplication
- Added Optional[] to optional outputs and types
- removed duplication of image = context.services.images.get_pil_images(self.image.image_name) declaration
- Still need to find a way to deal with mask_pil None typing errors
* face(facetools): fix typing issues, add validation, clean up structure
* feat(facetools): update field descriptions
* Update FaceOff_FaceScale2x.json
- update FaceOff workflow after Bounded Image field removed in place of inheriting Image out field from ImageOutput
* feat(facetools): pass through original image on facemask if invalid face ids requested
* feat(facetools): tidy variable names & fn calls
* feat(facetools): bundle inter font, draw ids with it
Inter is a SIL Open Font license. The license is included and is fully permissive. Inter is the same font the UI and commercial application already uses.
Only the "regular" version is bundled.
* chore(facetools): isort & fix mypy issues
* docs(facetools): update and format docs
---------
Co-authored-by: Millun Atluri <millun.atluri@gmail.com>
Co-authored-by: Millun Atluri <Millu@users.noreply.github.com>
Co-authored-by: psychedelicious <4822129+psychedelicious@users.noreply.github.com>
All invocation metadata (type, title, tags and category) are now defined in decorators.
The decorators add the `type: Literal["invocation_type"]: "invocation_type"` field to the invocation.
Category is a new invocation metadata, but it is not used by the frontend just yet.
- `@invocation()` decorator for invocations
```py
@invocation(
"sdxl_compel_prompt",
title="SDXL Prompt",
tags=["sdxl", "compel", "prompt"],
category="conditioning",
)
class SDXLCompelPromptInvocation(BaseInvocation, SDXLPromptInvocationBase):
...
```
- `@invocation_output()` decorator for invocation outputs
```py
@invocation_output("clip_skip_output")
class ClipSkipInvocationOutput(BaseInvocationOutput):
...
```
- update invocation docs
- add category to decorator
- regen frontend types
author Kyle Schouviller <kyle0654@hotmail.com> 1669872800 -0800
committer Kyle Schouviller <kyle0654@hotmail.com> 1676240900 -0800
Adding base node architecture
Fix type annotation errors
Runs and generates, but breaks in saving session
Fix default model value setting. Fix deprecation warning.
Fixed node api
Adding markdown docs
Simplifying Generate construction in apps
[nodes] A few minor changes (#2510)
* Pin api-related requirements
* Remove confusing extra CORS origins list
* Adds response models for HTTP 200
[nodes] Adding graph_execution_state to soon replace session. Adding tests with pytest.
Minor typing fixes
[nodes] Fix some small output query hookups
[node] Fixing some additional typing issues
[nodes] Move and expand graph code. Add base item storage and sqlite implementation.
Update startup to match new code
[nodes] Add callbacks to item storage
[nodes] Adding an InvocationContext object to use for invocations to provide easier extensibility
[nodes] New execution model that handles iteration
[nodes] Fixing the CLI
[nodes] Adding a note to the CLI
[nodes] Split processing thread into separate service
[node] Add error message on node processing failure
Removing old files and duplicated packages
Adding python-multipart
