Commit Graph

140 Commits

Author SHA1 Message Date
Lincoln Stein
d968c6f379 refactor multifile download code 2024-05-17 22:29:19 -04:00
Lincoln Stein
2dae5eb7ad more refactoring; HF subfolders not working 2024-05-16 22:26:18 -04:00
Lincoln Stein
911a24479b add tests for model install file size reporting 2024-05-16 07:18:33 -04:00
Lincoln Stein
f29c406fed refactor model_install to work with refactored download queue 2024-05-13 22:49:15 -04:00
Lincoln Stein
0bf14c2830 add multifile_download() method to download service 2024-05-12 20:14:00 -06:00
Lincoln Stein
b48d4a049d bad implementation of diffusers folder download 2024-05-08 21:21:01 -07:00
Lincoln Stein
f211c95dbc move access token regex matching into download queue 2024-05-05 21:00:31 -04:00
Lincoln Stein
bb04f496e0 Merge branch 'main' into lstein/feat/simple-mm2-api 2024-04-28 11:33:26 -04:00
Lincoln Stein
70903ef057 refactor load_ckpt_from_url() 2024-04-28 11:33:23 -04:00
Lincoln Stein
d72f272f16 Address change requests in first round of PR reviews.
Pending:

- Move model install calls into model manager and create passthrus in invocation_context.
- Consider splitting load_model_from_url() into a call to get the path and a call to load the path.
2024-04-24 23:53:30 -04:00
Lincoln Stein
2b9f06dc4c
Re-enable app shutdown actions (#6244)
* closes #6242

* only override sigINT during slow model scanning

* fix ruff formatting

---------

Co-authored-by: Lincoln Stein <lstein@gmail.com>
2024-04-19 06:45:42 -04:00
Lincoln Stein
34cdfc61ab
Merge branch 'main' into lstein/feat/simple-mm2-api 2024-04-17 17:18:13 -04:00
Lincoln Stein
fce6b3e44c maybe solve race issue 2024-04-16 13:09:26 +10:00
Lincoln Stein
470a39935c fix merge conflicts with main 2024-04-15 09:24:57 -04:00
Lincoln Stein
e93f4d632d
[util] Add generic torch device class (#6174)
* introduce new abstraction layer for GPU devices

* add unit test for device abstraction

* fix ruff

* convert TorchDeviceSelect into a stateless class

* move logic to select context-specific execution device into context API

* add mock hardware environments to pytest

* remove dangling mocker fixture

* fix unit test for running on non-CUDA systems

* remove unimplemented get_execution_device() call

* remove autocast precision

* Multiple changes:

1. Remove TorchDeviceSelect.get_execution_device(), as well as calls to
   context.models.get_execution_device().
2. Rename TorchDeviceSelect to TorchDevice
3. Added back the legacy public API defined in `invocation_api`, including
   choose_precision().
4. Added a config file migration script to accommodate removal of precision=autocast.

* add deprecation warnings to choose_torch_device() and choose_precision()

* fix test crash

* remove app_config argument from choose_torch_device() and choose_torch_dtype()

---------

Co-authored-by: Lincoln Stein <lstein@gmail.com>
2024-04-15 13:12:49 +00:00
Lincoln Stein
3ddd7ced49 change names of convert and download caches and add migration script 2024-04-14 15:57:33 -04:00
psychedelicious
8c15d14099 fix: use locale encoding
We have had a few bugs with v4 related to file encodings, especially on Windows.

Windows uses its own character encodings instead of `utf-8`, often `cp1252`. Some characters cannot be decoded using `utf-8`, causing `UnicodeDecodeError`.

There are a couple places where this can cause problems:
- In the installer bootstrap, we install or upgrade `pip` and decode the result, using `subprocess`.

  The input to this includes the user's home dir. In #6105, the user had one of the problematic characters in their username. `subprocess` attempts and fails to decode the username, which crashes the installer.

  To fix this, we need to use `locale.getpreferredencoding()` when executing the command.
- Similarly, in the model install service and config class, we attempt to load a yaml config file. If a problematic character is in the path to the file (which often includes the user's home dir), we can get the same error.

  One example is  #6129 in which the models.yaml migration fails.

  To fix this, we need to open the file with `locale.getpreferredencoding()`.
2024-04-04 15:30:47 +11:00
psychedelicious
7ff2371c07 fix(mm): do not rename model file if model record is renamed
Renaming the model file to the model name introduces unnecessary contraints on model names.

For example, a model name can technically be any length, but a model _filename_ cannot be too long.

There are also constraints on valid characters for filenames which shouldn't be applied to model record names.

I believe the old behaviour is a holdover from the old system.
2024-04-04 07:17:38 +11:00
psychedelicious
b0ffe36d21 feat(mm): update v3 models.yaml migration logic to handle relative paths for legacy config files 2024-03-29 10:56:06 -04:00
psychedelicious
6b3fdb8a93 fix(mm): handle relative model paths in _register_orphaned_models 2024-03-29 10:56:06 -04:00
psychedelicious
6d261a5a13 fix(mm): handle relative conversion config paths
I have tested main, controlnet and vae checkpoint conversions.
2024-03-29 10:56:06 -04:00
psychedelicious
c5d1bd1360 feat(mm): use relative paths for invoke-managed models
We switched all model paths to be absolute in #5900. In hindsight, this is a mistake, because it makes the `models_dir` non-portable.

This change reverts to the previous model pathing:
- Invoke-managed models (in the `models_dir`) are stored with relative paths
- Non-invoke-managed models (outside the `models_dir`, i.e. in-place installed models) still have absolute paths.

## Why absolute paths make things non-portable

Let's say my `models_dir` is `/media/rhino/invokeai/models/`. In the DB, all model paths will be absolute children of this path, like this:

- `/media/rhino/invokeai/models/sd-1/main/model1.ckpt`

I want to change my `models_dir` to `/home/bat/invokeai/models/`. I update my `invokeai.yaml` file and physically move the files to that directory.

On startup, the app checks for missing models. Because all of my model paths were absolute, they now point to a nonexistent path. All models are broken.

There are a couple options to recover from this situation, neither of which are reasonable:

1. The user must manually update every model's path. Unacceptable UX.
2. On startup, we check for missing models. For each missing model, we compare its path with the last-known models dir. If there is a match, we replace that portion of the path with the new models dir. Then we re-check to see if the path exists. If it does, we update the models DB entry. Brittle and requires a new DB entry for last-known models dir.

It's better to use relative paths for Invoke-managed models.
2024-03-29 10:56:06 -04:00
psychedelicious
73c326680a feat(mm): remove autoimport; revise startup model scanning
These two changes are interrelated.

## Autoimport

The autoimport feature can be easily replicated using the scan folder tab in the model manager. Removing the implicit autoimport reduces surface area and unifies all model installation into the UI.

This functionality is removed, and the `autoimport_dir` config setting is removed.

## Startup model dir scanning

We scanned the invoke-managed models dir on startup and took certain actions:

- Register orphaned model files
- Remove model records from the db when the model path doesn't exist

### Orphaned model files

We should never have orphaned model files during normal use - we manage the models directory, and we only delete files when the user requests it.

During testing or development, when a fresh DB or memory DB is used, we could end up with orphaned models that should be registered.

Instead of always scanning for orphaned models and registering them, we now only do the scan if the new `scan_models_on_startup` config flag is set.

The description for this setting indicates it is intended for use for testing only.

### Remove records for missing model files

This functionality could unexpectedly wipe models from the db.

For example, if your models dir was on external media, and that media was inaccessible during startup, the scan would see all your models as missing and delete them from the db.

The "proactive" scan is removed. Instead, we will scan for missing models and log a warning if we find a model whose path doesn't exist. No possibility for data loss.
2024-03-28 12:35:41 +11:00
psychedelicious
16dad07294 feat(mm): improved install error log message in terminal 2024-03-27 08:34:00 +11:00
psychedelicious
b2ea749c72 fix(mm): handle any error during installation
Previously we only handled expected error types. If a different error was raised, the install job would end up in an unexpected state where it has failed and isn't doing anything, but its status is still running.

This indirectly prevents the installer threads from exiting - they are waiting for all jobs to be completed, including the failed-but-still-running job.

We need to handle any error here to prevent this.
2024-03-27 08:34:00 +11:00
Lincoln Stein
0f02a72cb9 allow deletion of symlinked models in models dir 2024-03-22 18:29:24 -07:00
Lincoln Stein
9cbf78542c remove dangling comment 2024-03-22 16:35:42 +11:00
Lincoln Stein
34f5259980 catch ^C at startup time while models are being scanned 2024-03-22 16:35:42 +11:00
Lincoln Stein
eb558d72d8
Fix minor bugs involving model manager handling of model paths (#6024)
* Fix minor bugs involving model manager handling of model paths

- Leave models found in the `autoimport` directory there. Do not move them
  into the `models` hierarchy.
- If model name, type or base is updated and model is in the `models` directory,
  update its path as appropriate.
- On startup during model scanning, if a model's path is a symbolic link, then resolve
  to an absolute path before deciding it is a new model that must be hashed and
  registered. (This prevents needless hashing at startup time).

* fix issue with dropped suffix

---------

Co-authored-by: Lincoln Stein <lstein@gmail.com>
2024-03-22 01:14:45 +00:00
psychedelicious
d0a936ebd4 fix(mm): do not write config file when migrating models.yaml 2024-03-22 09:53:02 +11:00
psychedelicious
75f4e27522 tidy(mm): clean up model download/install logs 2024-03-21 16:41:20 +11:00
psychedelicious
8ae757334e feat(mm): make installer thread logging stmts debug 2024-03-21 16:41:20 +11:00
Lincoln Stein
689cb9d31d after stopping install and download services, wait for thread exit 2024-03-21 16:41:20 +11:00
Lincoln Stein
0cab1d1e04 added debugging statements 2024-03-21 16:41:20 +11:00
Lincoln Stein
9bd7dabed3 refactor big _install_next_item() loop 2024-03-21 16:41:20 +11:00
psychedelicious
f5337c7ce2 fix(config): handle relative paths to v3 models.yamls 2024-03-20 15:05:25 +11:00
psychedelicious
6c13fa13ea fix(mm): regression from change to legacy conf dir change 2024-03-20 15:05:25 +11:00
psychedelicious
6c558279dd feat(config): add CLI arg to specify config file
This allows users to create simple "profiles" via separate `invokeai.yaml` files.

- Remove `InvokeAIAppConfig.set_root()`, it's extraneous
- Remove `InvokeAIAppConfig.merge_from_file()`, it's extraneous
- Add `--config` to the app arg parser, add `InvokeAIAppConfig._config_file`, and consume in the config singleton getter
- `InvokeAIAppConfig.init_file_path` -> `InvokeAIAppConfig.config_file_path`
2024-03-20 15:05:25 +11:00
psychedelicious
429f87c60b fix(mm): HFModelSource string format
The dunder `__str__` method for `HFModelSource` was appending a colon `:` to the end of the source strings.
2024-03-20 15:05:25 +11:00
Lincoln Stein
74a51571a0
Fix race condition causing hangs during model install unit tests (#5994)
* fix race condition causing hangs during model install unit tests

* remove extraneous sanity checks

---------

Co-authored-by: Lincoln Stein <lstein@gmail.com>
2024-03-19 20:54:49 +00:00
psychedelicious
e76cc71e81 fix(config): edge cases in models.yaml migration
When running the configurator, the `legacy_models_conf_path` was stripped when saving the config file. Then the migration logic didn't fire correctly, and the custom models.yaml paths weren't migrated into the db.

- Rework the logic to migrate this path by adding it to the config object as a normal field that is not excluded from serialization.
- Rearrange the models.yaml migration logic to remove the legacy path after migrating, then write the config file. This way, the legacy path doesn't stick around.
- Move the schema version into the config object.
- Back up the config file before attempting migration.
- Add tests to cover this edge case
2024-03-19 09:24:28 +11:00
psychedelicious
e8b030427d fix(config): do not discard conf_path, migrate custom models.yaml
Hold onto `conf_path` temporarily while migrating `invokeai.yaml` so that it gets migrated correctly as the model installer starts up. Stashed as `legacy_models_yaml_path` in the config, excluded from serialization.
2024-03-19 09:24:28 +11:00
psychedelicious
897fe497dc fix(config): use new get_config across the app, use correct settings 2024-03-19 09:24:28 +11:00
psychedelicious
4df28f1de6 fix(config): use yaml module instead of omegaconf when migrating models.yaml
Also use new paths.
2024-03-19 09:24:28 +11:00
Brandon Rising
ea5bc94b9c Resolve when instantiating _cached_model_paths 2024-03-18 11:17:23 +11:00
Brandon Rising
a1743647b7 Stop registering and moving models which have symlinks in the models dir 2024-03-18 11:17:23 +11:00
Lincoln Stein
71a1740740 Remove core safetensors->diffusers conversion models
- No longer install core conversion models. Use the HuggingFace cache to load
  them if and when needed.

- Call directly into the diffusers library to perform conversions with only shallow
   wrappers around them to massage arguments, etc.

- At root configuration time, do not create all the possible model subdirectories,
  but let them be created and populated at model install time.

- Remove checks for missing core conversion files, since they are no
  longer installed.
2024-03-17 19:13:18 -04:00
Lincoln Stein
a0420d1442 fix ruff error 2024-03-17 14:01:04 -04:00
Lincoln Stein
a17021ba0c allow removal of models with legacy relative path addressing 2024-03-17 09:58:16 -04:00
psychedelicious
eb6e6548ed feat(mm): faster hashing for spinning disk HDDs
BLAKE3 has poor performance on spinning disks when parallelized. See https://github.com/BLAKE3-team/BLAKE3/issues/31

- Replace `skip_model_hash` setting with `hashing_algorithm`. Any algorithm we support is accepted.
- Add `random` algorithm: hashes a UUID with BLAKE3 to create a random "hash". Equivalent to the previous skip functionality.
- Add `blake3_single` algorithm: hashes on a single thread using BLAKE3, fixes the aforementioned performance issue
- Update model probe to accept the algorithm to hash with as an optional arg, defaulting to `blake3`
- Update all calls of the probe to use the app's configured hashing algorithm
- Update an external script that probes models
- Update tests
- Move ModelHash into its own module to avoid circuclar import issues
2024-03-14 15:54:42 +11:00