Commit Graph

8908 Commits

Author SHA1 Message Date
skunkworxdark
c84526fae5 Fixed Tests that where using round_to_8 and removed redundant tests 2023-12-11 17:05:45 +00:00
skunkworxdark
f762940335 Merge branch 'main' into tiled-upscaling-graph 2023-12-11 16:57:36 +00:00
skunkworxdark
fefb78795f - Even_spilt overlap renamed to overlap_fraction
- min_overlap removed * restrictions and round_to_8
- min_overlap handles tile size > image size by clipping the num tiles to 1.
- Updated assert test on min_overlap.
2023-12-11 16:55:27 +00:00
psychedelicious
ef8284f009 fix(db): fix tests 2023-12-11 16:41:47 +11:00
psychedelicious
290851016e feat(db): move sqlite_migrator into its own module 2023-12-11 16:41:30 +11:00
psychedelicious
fa7d002175 fix(tests): fix typing issues 2023-12-11 16:22:29 +11:00
psychedelicious@windows
f1b6f78319 fix(db): fix windows db migrator tests
- Ensure db files are closed before manipulating them
- Use contextlib.closing() so that sqlite connections are closed on existing the context
2023-12-11 16:14:25 +11:00
psychedelicious
26ab917021 fix(tests): add sqlite migrator to test fixtures 2023-12-11 16:14:25 +11:00
psychedelicious
4f3c32a2ee fix(db): remove errant print stmts 2023-12-11 16:14:25 +11:00
psychedelicious
77065b1ce1 feat(db): update test for migration chain for missing from 0 2023-12-11 16:14:25 +11:00
psychedelicious
41db92b9e8 feat(db): add check for missing migration from 0 2023-12-11 16:14:25 +11:00
psychedelicious
c823f5667b feat(db): update sqlite migrator tests 2023-12-11 16:14:25 +11:00
psychedelicious
3227b30430 feat(db): extract non-stateful logic to class methods 2023-12-11 16:14:25 +11:00
psychedelicious
567f107a81 feat(db): return backup_db_path, move log stmt to run_migrations 2023-12-11 16:14:25 +11:00
psychedelicious
b3d5955bc7 fix(db): rename Migrator._migrations -> _migration_set 2023-12-11 16:14:25 +11:00
psychedelicious
8726b203d4 fix(db): fix migration chain validation 2023-12-11 16:14:25 +11:00
psychedelicious
b3f92e0547 fix(db): fix docstring 2023-12-11 16:14:25 +11:00
psychedelicious
72c9a7663f fix(db): add docstring 2023-12-11 16:14:25 +11:00
psychedelicious
fcb9e89bd7 feat(db): tidy db naming utils 2023-12-11 16:14:25 +11:00
psychedelicious
56966d6d05 feat(db): only reinit db if migrations occurred 2023-12-11 16:14:25 +11:00
psychedelicious
e46dc9b34e fix(db): close db conn before reinitializing 2023-12-11 16:14:25 +11:00
psychedelicious
e461f9925e feat(db): invert backup/restore logic
Do the migration on a temp copy of the db, then back up the original and move the temp into its file.
2023-12-11 16:14:25 +11:00
psychedelicious
abeb1bd3b3 feat(db): reduce power MigrateCallback, only gets cursor
use partial to provide extra dependencies for the image workflow migration function
2023-12-11 16:14:25 +11:00
psychedelicious
83e820d721 feat(db): decouple from SqliteDatabase 2023-12-11 16:14:25 +11:00
psychedelicious
f8e4b93a74 feat(db): add migration lock file 2023-12-11 16:14:25 +11:00
psychedelicious
0710ec30cf feat(db): incorporate feedback 2023-12-11 16:14:25 +11:00
psychedelicious
c382329e8c feat(db): move migrator out of SqliteDatabase 2023-12-11 16:14:25 +11:00
psychedelicious
a2dc780188 feat: add script to migrate image workflows 2023-12-11 16:14:25 +11:00
psychedelicious@windows
abc9dc4d17 fix(tests): fix sqlite migrator backup and restore test
On Windows, we must ensure the connection to the database is closed before exiting the tempfile context.

Also, rejiggered the thing to use the file directly.
2023-12-11 16:14:25 +11:00
psychedelicious
3c692018cd fix(db): make idempotency test actually test something 2023-12-11 16:14:25 +11:00
psychedelicious
3ba3c1918c fix(db): remove duplicated test case 2023-12-11 16:14:25 +11:00
psychedelicious
f2c6819d68 feat(db): add SQLiteMigrator to perform db migrations 2023-12-11 16:14:25 +11:00
Kent Keirsey
ef807cf63a
Refactor model manager: model installer component (#5171)
## What type of PR is this? (check all applicable)

- [X] Refactor
- [X] Feature
- [ ] Bug Fix
- [ ] Optimization
- [X] Documentation Update
- [ ] Community Node Submission


## Have you discussed this change with the InvokeAI team?
- [X] Yes
- [ ] No, because:

      
## Have you updated all relevant documentation?
- [X] Yes
- [ ] No


## Description

This is the next phase of the model manager refactor, as discussed with
@psychedelicious and @RyanJDick. This implements the model installer,
which is responsible for managing model weights on disk and installing
new models.

Currently only installation of local files and directories is supported.
Remote installation will be implemented after the queued download
manager is reviewed and approved.

Please see the documentation located at
[docs/contributing/MODEL_MANAGER.md](8695ad6f59/docs/contributing/MODEL_MANAGER.md (model-installation))
for an explanation of how this module works.

Things that have changed relative to the current implementation.

1. Model importation runs in a background thread. Access to the
installation status is through a ModelInstallJob object returned by the
`import_model()` call. In addition, the installation process generates a
series of `model_install` events on the event bus.
2. `model_install_progress` events are documented, but not currently
issued. These will be issued when background downloading is implemented.
3. The model installer currently runs in parallel to the current model
manager. The frontend continues to use `configs/models.yaml` and ignores
what is in the `model_config` table of `invokeai.db`.
4. When the installer is initialized at app startup time, it
synchronizes its database to the contents of the InvokeAI `models`
directory. The current model manager does this as well, so you will see
two log messages indicating that this directory is being scanned.


## Related Tickets & Documents

<!--
For pull requests that relate or close an issue, please include them
below. 

For example having the text: "closes #1234" would connect the current
pull
request to issue 1234.  And when we merge the pull request, Github will
automatically close the issue.
-->

- Related Issue #
- Closes #

## QA Instructions, Screenshots, Recordings

You can test using the FastAPI swagger pages at
http://localhost:9090/docs. Use the calls listed under
`model_manager_v2`. Be aware that only installation of local models
(indicated by their file or directory path) are currently supported.

## Added/updated tests?

- [X] Yes -- see
`tests/app/services/model_install/test_model_install.py`
- [ ] No : _please replace this line with details on why tests
      have not been included_

## [optional] Are there any post deployment tasks we need to perform?
2023-12-10 23:16:39 -05:00
Lincoln Stein
bbcd58e681 Merge branch 'refactor/model-manager-3' of github.com:invoke-ai/InvokeAI into refactor/model-manager-3 2023-12-10 21:34:14 -05:00
Lincoln Stein
36043bf38b fixed docstring in probe module 2023-12-10 21:33:54 -05:00
Lincoln Stein
fd68c47920
Merge branch 'main' into refactor/model-manager-3 2023-12-10 21:26:44 -05:00
psychedelicious
c5c975c7a9 fix(installer): fix exit on new version 2023-12-11 12:30:13 +11:00
psychedelicious
41ad13c282 feat(installer): do not print when aliasing python
Potentially confusing and not useful
2023-12-11 12:30:13 +11:00
psychedelicious
e9d7e6bdd5 feat(installer): make active venv error red instead of yellow 2023-12-11 12:30:13 +11:00
psychedelicious
49b74d189e feat(installer): improve messages, simplify script
- Color outputs
- Clarify messages
- Do not offer to use existing frontend build (insurance - prevents accidentally using old build)
2023-12-11 12:30:13 +11:00
psychedelicious
179bc64490 feat(create_installer): remove extraneous conditional
Using `-f` is functionally equivalent to first checking if the dir exists before removing it. We just want to ensure the build dir doesn't exists.
2023-12-11 12:30:13 +11:00
psychedelicious
1feab3da37 fix(installer): update msg in create_installer
More accurate/clearer messages
2023-12-11 12:30:13 +11:00
psychedelicious
0a15f3fc35 fix(tests): remove test for frontend build 2023-12-11 12:30:13 +11:00
psychedelicious
daf00efa4d fix(api): only attempt to serve UI build if it exists 2023-12-11 12:30:13 +11:00
psychedelicious
55cfb879d0 feat: no frontend build in repo
In other words, build frontend when creating installer.

Changes to `create_installer.sh`

- If `python` is not in `PATH` but `python3` is, alias them (well, via function). This is needed on some machines to run the installer without symlinking to `python3`.
- Make the messages about pushing tags clearer. The script force-pushes, so it's possible to accidentally take destructive action. I'm not sure how to otherwise prevent damage, so I just added a warning.
- Print out `pwd` when prompting about being in the `installer` dir.
- Rebuild the frontend - if there is already a frontend build, first checks if the user wants to rebuild it.
- Checks for existence of `../build` dir before deleting - if the dir doesn't exist, the script errors and exits at this point.
- Format and spell check.

Other changes:

- Ignore `dist/` folder.
- Delete `dist/`.

**Note: you may need to use `git rm --cached invokeai/app/frontend/web/dist/` if git still wants to track `dist/`.**
2023-12-11 12:30:13 +11:00
Lincoln Stein
de2879f602 port new code for detecting sdxl-based embeddings 2023-12-10 15:48:02 -05:00
Lincoln Stein
3b1ff4a7f4 resolve test failure caused by renamed sqlite_database module 2023-12-10 12:59:00 -05:00
Lincoln Stein
d7f7fbc8c2 Merge branch 'main' into refactor/model-manager-3 2023-12-10 12:55:28 -05:00
Lincoln Stein
e2567a7e31 Merge branch 'refactor/model-manager-3' of github.com:invoke-ai/InvokeAI into refactor/model-manager-3 2023-12-10 12:55:24 -05:00
Lincoln Stein
2f3457c02a rename installer __del__() to stop(). Improve probe error messages 2023-12-10 12:55:01 -05:00