Commit Graph

83 Commits

Author SHA1 Message Date
psychedelicious
9376b13435 fix(mm): models lose file extension when syncing
We were stripping the file extension from file models when  moving them in `_sync_model_path`. For example, `some_model.safetensors` would be moved to `some_model`, which of course breaks things.

Instead of using the model's name as the new path, use the model's path's last segment. This is the same behaviour for directories, but for files, it retains the file extension.
2024-03-10 13:36:09 +11:00
psychedelicious
eec82afd89 fix(mm): fix models.yaml backup filename
Was erroneously `models.bak`, now `models.yaml.bak`
2024-03-10 13:36:09 +11:00
psychedelicious
56e7c04475 tidy(mm): remove extraneous dependencies in model search
- `config` is unused
- `stats` is created on instantiation
- `logger` uses the app logger
2024-03-10 12:09:47 +11:00
psychedelicious
d4686b7f64 fix(mm): yaml migration fixup
- If the metadata yaml has an invalid version, exist the app. If we don't, the app will crawl the models dir and add models to the db without having first parsed `models.yaml`. This should not happen often, as the vast majority of users are on v3.0.0 models.yaml files.
- Fix off-by-one error with models count (need to pop the `__metadata__` stanza
- After a successful migration, rename `models.yaml` to `models.yaml.bak` to prevent the migration logic from re-running on subsequent app startups.
2024-03-09 08:37:45 -06:00
psychedelicious
67163c2224 fix(mm): only move model files if necessary
The old logic to check if a model needed to be moved relied on the model path being a relative path. Paths are now absolute, causing this check to fail. We then assumed the paths were different and moved the model from its current location to, well, its current location.

Use more resilient method to check if a model should be moved.
2024-03-09 22:58:26 +11:00
Brandon Rising
f01e81d382 Run ruff 2024-03-08 18:46:17 -05:00
maryhipp
a50e0a4802 use correct key name from yaml 2024-03-08 18:46:17 -05:00
maryhipp
df0a5aa92a pass config_path to migration path, make sure it uses absolute path 2024-03-08 18:46:17 -05:00
Brandon Rising
ee38fbe89c Remove check for models dir in model deletion, update tests to always assume the model path is an absolute path 2024-03-08 15:36:14 -05:00
Brandon Rising
6e2cef1db5 Remove instances making models relative to the model dir 2024-03-08 15:36:14 -05:00
Brandon Rising
b1f5ac4548 fix path 2024-03-08 15:36:14 -05:00
Brandon Rising
e52274ecac Experiment with using absolute paths within model management 2024-03-08 15:36:14 -05:00
maryhipp
d5674150fa ruff 2024-03-08 13:02:04 -05:00
maryhipp
0cb2cf6644 wrap version check in try/except 2024-03-08 13:02:04 -05:00
maryhipp
da87266c9c remove log 2024-03-08 13:02:04 -05:00
maryhipp
35731a6f51 fix null description, add logging 2024-03-08 13:02:04 -05:00
Brandon Rising
a3dfa161a8 Run ruff 2024-03-08 13:02:04 -05:00
Brandon Rising
42d606f07c use register instead of heuristic import, get rid of typing warnings 2024-03-08 13:02:04 -05:00
maryhipp
9063b1ae61 on model manager start, look to see if yaml needs to be migrated and do it if so 2024-03-08 13:02:04 -05:00
psychedelicious
576bb4a61d feat(mm): support generic API tokens via regex/token pairs in config
A list of regex and token pairs is accepted. As a file is downloaded by the model installer, the URL is tested against the provided regex/token pairs. The token for the first matching regex is used during download, added as a bearer token.
2024-03-08 13:59:59 +11:00
Brandon Rising
04229f4a21 Run ruff 2024-03-08 13:59:59 +11:00
Brandon Rising
73a190fb6e Add remote_repo_api_key config to be added as a token query param for all remote url model downloads 2024-03-08 13:59:59 +11:00
Brandon Rising
952d97741e Remove civit ai from tests and documentation 2024-03-08 13:59:59 +11:00
Brandon Rising
d1f859a446 Remove civit AI model install resources 2024-03-08 13:59:59 +11:00
psychedelicious
bcf58cac59 feat(mm): add config to skip model hash
This is useful for when you are using a memory DB and do not want to wait for all models to be hashed on startup.
2024-03-05 23:50:19 +11:00
psychedelicious
e8797787cf fix(mm): fix incorrect calls to update_model 2024-03-05 23:50:19 +11:00
psychedelicious
656839fcd1 fix(mm): fix typing on heuristic_import 2024-03-05 23:50:19 +11:00
psychedelicious
78895b3e80 fix(mm): add missing inplace parameter to model install abc 2024-03-05 23:50:19 +11:00
psychedelicious
c953e61294 tidy(mm): "trigger_words" -> "trigger_phrases" 2024-03-05 23:50:19 +11:00
psychedelicious
44c40d7d1a refactor(mm): remove unused metadata logic, fix tests
- Metadata is merged with the config. We can simplify the MM substantially and remove the handling for metadata.
- Per discussion, we don't have an ETA for frontend implementation of tags, and with the realization that the tags from CivitAI are largely useless, there's no reason to keep tags in the MM right now. When we are ready to implement tags on the frontend, we can refer back to the implementation here and use it if it supports the design.
- Fix all tests.
2024-03-05 23:50:19 +11:00
psychedelicious
c3aa985c93 refactor(mm): get metadata working 2024-03-05 23:50:19 +11:00
psychedelicious
7cb0da1f66 refactor(mm): wip schema changes 2024-03-05 23:50:19 +11:00
psychedelicious
9378e47a06 feat(mm): add source_type to model configs 2024-03-05 23:50:19 +11:00
psychedelicious
9b40c28144 tidy(mm): rename ckpy "config" -> "config_path" 2024-03-05 23:50:19 +11:00
psychedelicious
76cbc745e1 refactor(mm): add CheckpointConfigBase for all ckpt models 2024-03-05 23:50:19 +11:00
psychedelicious
b880a31039 refactor(mm): remove ztsnr_training field on _MainConfig
This is used to determine the CFG Rescale Multiplier setting. We'll handle this in the UI as a default setting.
2024-03-05 23:50:19 +11:00
Brandon Rising
893bcd16fc Next: Allow in place local installs of models 2024-03-04 23:11:41 +11:00
psychedelicious
ae99428883 fix(mm): use UUIDv4 for key
This changes the functionality of this PR to only use the updated hashing for model hashes with a UUID for the key.
2024-03-03 14:32:14 +11:00
Lincoln Stein
a72056e0df make model key assignment deterministic
- When installing, model keys are now calculated from the model contents.
- .safetensors, .ckpt and other single file models are hashed with sha1
- The contents of diffusers directories are hashed using imohash (faster)

fixup yaml->sql db migration script to assign deterministic key

- this commit also detects and assigns the correct image encoder for
  ip adapter models.
2024-03-03 14:32:14 +11:00
Lincoln Stein
0832e1818e Fix problem of all installed models being assigned "<NOKEY>"
- Also fix redundant scanning of models directory at startup.
2024-02-29 22:31:05 -05:00
Lincoln Stein
813a086cfe fix race condition between downloading last file and starting install 2024-03-01 10:42:33 +11:00
Lincoln Stein
e18533e3b5 add debugging statements and a timeout to download test 2024-03-01 10:42:33 +11:00
psychedelicious
12e859835b feat(mm): add log stmt for download complete event 2024-03-01 10:42:33 +11:00
psychedelicious
ea7b7bcf40 chore: ruff 2024-03-01 10:42:33 +11:00
psychedelicious
e22c4987bf chore: ruff 2024-03-01 10:42:33 +11:00
maryhipp
26a209a00d add error_reason to ModelInstallJob 2024-03-01 10:42:33 +11:00
Jennifer Player
ea364bdf82 delete model imports and prune all finished, update state with socket messages 2024-03-01 10:42:33 +11:00
Brandon Rising
c778ab8db4 Allow passing in key on register 2024-03-01 10:42:33 +11:00
Brandon Rising
65b91356d0 Remove passing keys in on register 2024-03-01 10:42:33 +11:00
Brandon Rising
008716040b Allow users to run model manager without cuda 2024-03-01 10:42:33 +11:00