Commit Graph

8938 Commits

Author SHA1 Message Date
Lincoln Stein
22ccaa4e9a
[Feature] Allow the model record migrate script to update existing model records (#5264)
## What type of PR is this? (check all applicable)

- [ ] Refactor
- [X] Feature
- [ ] Bug Fix
- [ ] Optimization
- [ ] Documentation Update
- [ ] Community Node Submission


## Have you discussed this change with the InvokeAI team?
- [X] Yes
- [ ] No, because:

      
## Have you updated all relevant documentation?
- [ ] Yes
- [X] No


## Description

1. The new model manager sqlite3-based configuration record storage
system is automatically populated with probed values from existing
models found in the models path when `invokeai-web` starts up for the
first time. However, the user's customization of these models in
`invokeai.yaml`, including such things as the prediction type and model
description, are not automatically copied over. This PR enhances the
`invokeai-migrate-models-to-db` script so that any customized
configuration data from `invokeai.yaml` replaces the original probed
values. This script only needs to be run once, but it does not hurt to
run it additional times. In the near future, I'm going to register this
module with psychedelicious's sqlite migration system so that the update
happens automatically during database migration.

2. The SQL-based model config record system stores a JSON version of the
config, as well as several fields that are broken out into individual
columns for search/indexing purposes. This PR keeps the JSON and the
broken-out fields in sync using the `json_extract()` sqlite3 function to
populate the broken out `base`, `type`, `name`, `path` and `format`
fields in the `model_config` table.

3. Finally, this PR fixes the annoying `invokeai-web` shutdown message:
`TypeError: ModelInstallService.stop() takes 1 positional argument but 2
were given`

## Related Tickets & Documents


- Related Issue #
- Closes #

## QA Instructions, Screenshots, Recordings

If you've run `invokeai-web` at any time since PR #5039, your
`invokeai.db` will have a `model_config` table containing probe
information from all models in the invokeai models directory as well as
those in `autoimport` (if applicable). However, any models present in
`models.yaml` whose paths are outside these directories will not be
present. To add them, and to update the description and other values
from `models.yaml`, run the command `invokeai-migrate-models-to-db`. You
should see the missing models added to the database table with the
correct information.

<!-- 
Please provide steps on how to test changes, any hardware or 
software specifications as well as any other pertinent information. 
-->

## Added/updated tests?

- [X] Yes
- [ ] No : _please replace this line with details on why tests
      have not been included_

## [optional] Are there any post deployment tasks we need to perform?
2023-12-12 00:25:05 -05:00
Lincoln Stein
d277bd3c38
Merge branch 'main' into feat/enhance-model-db-migrate-script 2023-12-12 00:24:43 -05:00
Kevin Turner
fd4e041e7c feat: serve HTTPS when configured with ssl_certfile 2023-12-12 16:01:43 +11:00
Lincoln Stein
15a3e8076f
Merge branch 'main' into feat/enhance-model-db-migrate-script 2023-12-11 23:10:04 -05:00
Lincoln Stein
2fbe3a3104 fix ruff error 2023-12-11 23:04:18 -05:00
Lincoln Stein
b0cfa58526 allow the model record migrate script to update existing model records 2023-12-11 22:47:19 -05:00
Lincoln Stein
285ed26edd
Add commands to Makefile for convenient release preparation (#5263)
## What type of PR is this? (check all applicable)

- [ ] Refactor
- [X] Feature
- [ ] Bug Fix
- [ ] Optimization
- [ ] Documentation Update
- [ ] Community Node Submission


## Have you discussed this change with the InvokeAI team?
- [X] Yes
- [ ] No, because:

      
## Have you updated all relevant documentation?
- [X] Yes
- [ ] No


## Description

This PR does three things:

1) It separates out the script that creates the installer zipfile
(`create_installer.sh`) from the script that tags the repository with
the current release version (now called `tag_release.sh`)

2) It adds new targets to Makefile for running the installer script and
tagging.

3) It adds a `help` target that lists the Makefile targets:

```
$ make help
Developer commands:

ruff           Run ruff, fixing any safely-fixable errors and formatting
ruff-unsafe    Run ruff, fixing all fixable errors and formatting
mypy           Run mypy using the config in pyproject.toml to identify type mismatches and other coding errors
mypy-all       Run mypy ignoring the config in pyproject.tom but still ignoring missing imports
frontend-build Build the frontend in order to run on localhost:9090
frontend-dev   Run the frontend in developer mode on localhost:5173
installer-zip  Build the installer .zip file for the current version
tag-release    Tag the GitHub repository with the current version (use at release time only!)
```
`help` is also the default target so that the help message will print
out when only `make` is issued.

## Related Tickets & Documents

<!--
For pull requests that relate or close an issue, please include them
below. 

For example having the text: "closes #1234" would connect the current
pull
request to issue 1234.  And when we merge the pull request, Github will
automatically close the issue.
-->

- Related Issue #
- Closes #

## QA Instructions, Screenshots, Recordings

<!-- 
Please provide steps on how to test changes, any hardware or 
software specifications as well as any other pertinent information. 
-->

## Added/updated tests?

- [ ] Yes
- [X] No: not needed

## [optional] Are there any post deployment tasks we need to perform?
2023-12-11 22:33:45 -05:00
Lincoln Stein
02565b9a00
Merge branch 'main' into install/release-tools 2023-12-11 22:32:28 -05:00
Kent Keirsey
78a6024d6c
Tiled upscaling graph - new nodes (#5234)
## What type of PR is this? (check all applicable)

- [ ] Refactor
- [x] Feature
- [ ] Bug Fix
- [ ] Optimization
- [ ] Documentation Update
- [ ] Community Node Submission


## Have you discussed this change with the InvokeAI team?
- [x] Yes
- [ ] No, because:

      
## Have you updated all relevant documentation?
- [ ] Yes
- [x] No


## Description
Additional tile generation nodes of
CalculateImageTilesEvenSplitInvocation &
CalculateImageTilesMinimumOverlapInvocation
Additional blending method of merge_tiles_with_seam_blending
Updated Node MergeTilesToImageInvocation with seam blending

## Related Tickets & Documents

<!--
For pull requests that relate or close an issue, please include them
below. 

For example having the text: "closes #1234" would connect the current
pull
request to issue 1234.  And when we merge the pull request, Github will
automatically close the issue.
-->

- Related Issue #
- Closes #

## QA Instructions, Screenshots, Recordings

<!-- 
Please provide steps on how to test changes, any hardware or 
software specifications as well as any other pertinent information. 
-->

## Added/updated tests?

- [ ] Yes
- [ ] No : _please replace this line with details on why tests
      have not been included_

## [optional] Are there any post deployment tasks we need to perform?
2023-12-11 22:14:15 -05:00
psychedelicious@windows
95198da645 fix(db): fix sqlite migrator tests on windows 2023-12-12 13:54:47 +11:00
psychedelicious
ee1f1f3363 Merge remote-tracking branch 'origin/main' into feat/db/migrations 2023-12-12 13:39:47 +11:00
psychedelicious
18ba7feca1 feat(db): update docstrings 2023-12-12 13:35:46 +11:00
psychedelicious
55b0c7cdc9 feat(db): tidy migration_2 2023-12-12 13:30:29 +11:00
Lincoln Stein
713a83e7da
Merge branch 'main' into install/release-tools 2023-12-11 21:20:51 -05:00
Lincoln Stein
f3a97e06ec add the tag_release.sh script 2023-12-11 21:11:37 -05:00
psychedelicious
50815d36c6 feat(db): add tests for migration dependencies 2023-12-12 13:09:24 +11:00
psychedelicious
a69f518c76 feat(db): tidy dependencies for migrations 2023-12-12 13:09:09 +11:00
Lincoln Stein
18093c4f1d split installer zipfile script from tagging script; add make commands 2023-12-11 21:08:03 -05:00
psychedelicious
0cf7fe43af feat(db): refactor migrate callbacks to use dependencies, remote pre/post callbacks 2023-12-12 12:35:42 +11:00
psychedelicious
6063760ce2 feat(db): tweak docstring 2023-12-12 11:13:40 +11:00
psychedelicious
c5ba4f2ea5 feat(db): remove file backups
Instead of mucking about with the filesystem, we rely on SQLite transactions to handle failed migrations.
2023-12-12 11:12:46 +11:00
psychedelicious
3414437eea feat(db): instantiate SqliteMigrator with a SqliteDatabase
Simplifies a couple things:
- Init is more straightforward
- It's clear in the migrator that the connection we are working with is related to the SqliteDatabase
2023-12-12 10:46:08 +11:00
psychedelicious
417db71471 feat(db): decouple SqliteDatabase from config object
- Simplify init args to path (None means use memory), logger, and verbose
- Add docstrings to SqliteDatabase (it had almost none)
- Update all usages of the class
2023-12-12 10:30:37 +11:00
psychedelicious
afe4e55bf9 feat(db): simplify migration registration validation
With the previous change to assert that the to_version == from_version + 1, this validation can be simpler.
2023-12-12 09:52:03 +11:00
psychedelicious
55acc16b2d feat(db): require migration versions to be consecutive 2023-12-12 09:43:09 +11:00
Kent Keirsey
535ce10e99
Merge branch 'main' into tiled-upscaling-graph 2023-12-11 14:40:55 -05:00
Sam
11f4a48144 Add container GID 2023-12-11 14:30:40 -05:00
Sam
67ed4a0245 Respect CONTAINER_UID in Dockerfile chown
CONTAINER_UID is used for the user ID within the container, however I noticed the UID was hard coded to 1000 in the Dockerfile chown -R command.

This leaves the default as 1000, but allows it to be overrriden by setting CONTAINER_UID.
2023-12-11 14:30:40 -05:00
skunkworxdark
fbbc1037cd missed a rename of overlap to overlap_fraction in test for even_spilt 2023-12-11 17:23:28 +00:00
skunkworxdark
0852fd4e88 Updated tests for even_split overlap renamed to overlap_fraction 2023-12-11 17:17:29 +00:00
skunkworxdark
c84526fae5 Fixed Tests that where using round_to_8 and removed redundant tests 2023-12-11 17:05:45 +00:00
skunkworxdark
f762940335 Merge branch 'main' into tiled-upscaling-graph 2023-12-11 16:57:36 +00:00
skunkworxdark
fefb78795f - Even_spilt overlap renamed to overlap_fraction
- min_overlap removed * restrictions and round_to_8
- min_overlap handles tile size > image size by clipping the num tiles to 1.
- Updated assert test on min_overlap.
2023-12-11 16:55:27 +00:00
psychedelicious
ef8284f009 fix(db): fix tests 2023-12-11 16:41:47 +11:00
psychedelicious
290851016e feat(db): move sqlite_migrator into its own module 2023-12-11 16:41:30 +11:00
psychedelicious
fa7d002175 fix(tests): fix typing issues 2023-12-11 16:22:29 +11:00
psychedelicious@windows
f1b6f78319 fix(db): fix windows db migrator tests
- Ensure db files are closed before manipulating them
- Use contextlib.closing() so that sqlite connections are closed on existing the context
2023-12-11 16:14:25 +11:00
psychedelicious
26ab917021 fix(tests): add sqlite migrator to test fixtures 2023-12-11 16:14:25 +11:00
psychedelicious
4f3c32a2ee fix(db): remove errant print stmts 2023-12-11 16:14:25 +11:00
psychedelicious
77065b1ce1 feat(db): update test for migration chain for missing from 0 2023-12-11 16:14:25 +11:00
psychedelicious
41db92b9e8 feat(db): add check for missing migration from 0 2023-12-11 16:14:25 +11:00
psychedelicious
c823f5667b feat(db): update sqlite migrator tests 2023-12-11 16:14:25 +11:00
psychedelicious
3227b30430 feat(db): extract non-stateful logic to class methods 2023-12-11 16:14:25 +11:00
psychedelicious
567f107a81 feat(db): return backup_db_path, move log stmt to run_migrations 2023-12-11 16:14:25 +11:00
psychedelicious
b3d5955bc7 fix(db): rename Migrator._migrations -> _migration_set 2023-12-11 16:14:25 +11:00
psychedelicious
8726b203d4 fix(db): fix migration chain validation 2023-12-11 16:14:25 +11:00
psychedelicious
b3f92e0547 fix(db): fix docstring 2023-12-11 16:14:25 +11:00
psychedelicious
72c9a7663f fix(db): add docstring 2023-12-11 16:14:25 +11:00
psychedelicious
fcb9e89bd7 feat(db): tidy db naming utils 2023-12-11 16:14:25 +11:00
psychedelicious
56966d6d05 feat(db): only reinit db if migrations occurred 2023-12-11 16:14:25 +11:00