Jennifer Player
3a5314f1ca
install model if diffusers or single file, cleaned up backend logic to not mess with existing model install
2024-03-13 21:02:29 +11:00
Jennifer Player
4c0896e436
removed log
2024-03-13 21:02:29 +11:00
Jennifer Player
f7cd3cf1f4
added hf models import tab and route for getting available hf models
2024-03-13 21:02:29 +11:00
Jennifer Player
2a648da557
updated model manager to display when import item is cancelled
2024-03-13 09:18:05 +11:00
Brandon Rising
c454ccc65c
Run ruff
2024-03-11 15:53:00 -04:00
Brandon Rising
46fd3465ce
Skip list logic if the list only contains primitives
2024-03-11 15:53:00 -04:00
Brandon Rising
97afa6e2a6
Allow lists of basemodel objects in omegaconf
2024-03-11 15:53:00 -04:00
psychedelicious
9376b13435
fix(mm): models lose file extension when syncing
...
We were stripping the file extension from file models when moving them in `_sync_model_path`. For example, `some_model.safetensors` would be moved to `some_model`, which of course breaks things.
Instead of using the model's name as the new path, use the model's path's last segment. This is the same behaviour for directories, but for files, it retains the file extension.
2024-03-10 13:36:09 +11:00
psychedelicious
eec82afd89
fix(mm): fix models.yaml backup filename
...
Was erroneously `models.bak`, now `models.yaml.bak`
2024-03-10 13:36:09 +11:00
psychedelicious
56e7c04475
tidy(mm): remove extraneous dependencies in model search
...
- `config` is unused
- `stats` is created on instantiation
- `logger` uses the app logger
2024-03-10 12:09:47 +11:00
psychedelicious
92b0d13d0e
feat(nodes): "ModelField" -> "ModelIdentifierField", add hash/name/base/type
2024-03-10 11:03:38 +11:00
psychedelicious
5b51ebf1c4
docs: regenerate config docstrings
2024-03-10 10:38:52 +11:00
psychedelicious
59228643a9
docs: skip_model_hash -> model install category, use_memory_db -> development category
2024-03-10 10:38:52 +11:00
psychedelicious
b24657df11
docs: roll back adding examples to config docstrings
...
This isn't a valid docstring syntax and breaks the autogeneration
2024-03-10 10:38:52 +11:00
psychedelicious
d4686b7f64
fix(mm): yaml migration fixup
...
- If the metadata yaml has an invalid version, exist the app. If we don't, the app will crawl the models dir and add models to the db without having first parsed `models.yaml`. This should not happen often, as the vast majority of users are on v3.0.0 models.yaml files.
- Fix off-by-one error with models count (need to pop the `__metadata__` stanza
- After a successful migration, rename `models.yaml` to `models.yaml.bak` to prevent the migration logic from re-running on subsequent app startups.
2024-03-09 08:37:45 -06:00
psychedelicious
67163c2224
fix(mm): only move model files if necessary
...
The old logic to check if a model needed to be moved relied on the model path being a relative path. Paths are now absolute, causing this check to fail. We then assumed the paths were different and moved the model from its current location to, well, its current location.
Use more resilient method to check if a model should be moved.
2024-03-09 22:58:26 +11:00
Brandon Rising
f01e81d382
Run ruff
2024-03-08 18:46:17 -05:00
maryhipp
a50e0a4802
use correct key name from yaml
2024-03-08 18:46:17 -05:00
maryhipp
df0a5aa92a
pass config_path to migration path, make sure it uses absolute path
2024-03-08 18:46:17 -05:00
Brandon Rising
0bd9a0a9ea
Add ability to provide config examples in docs
2024-03-08 16:31:39 -05:00
Brandon Rising
4ae2cd242e
Update to include remote_api_tokens in the config docs
2024-03-08 16:31:39 -05:00
psychedelicious
deb1d4eb14
docs: run script to update config class's docstring
2024-03-08 16:31:39 -05:00
psychedelicious
eba1fc1355
docs: autogenerated app config docs
...
mkdocs can autogenerate python class docs from its docstrings. Our config is a pydantic model.
It's tedious and error-prone to duplicate docstrings from the pydantic field descriptions to the class docstrings.
- Add helper function to generate a mkdocs-compatible docstring from the InvokeAIAppConfig class fields
2024-03-08 16:31:39 -05:00
psychedelicious
96702c395e
feat(config): add deprecated category for config settings
...
It's not clear why these are still in the config class.
2024-03-08 16:31:39 -05:00
psychedelicious
3361aec065
docs(nodes): update config field descriptions
2024-03-08 16:31:39 -05:00
Brandon Rising
8ba4b2a150
Run ruff
2024-03-08 15:36:14 -05:00
Brandon Rising
df12e12e09
Run ruff
2024-03-08 15:36:14 -05:00
Brandon Rising
ee38fbe89c
Remove check for models dir in model deletion, update tests to always assume the model path is an absolute path
2024-03-08 15:36:14 -05:00
Brandon Rising
6e2cef1db5
Remove instances making models relative to the model dir
2024-03-08 15:36:14 -05:00
Brandon Rising
b1f5ac4548
fix path
2024-03-08 15:36:14 -05:00
Brandon Rising
e52274ecac
Experiment with using absolute paths within model management
2024-03-08 15:36:14 -05:00
maryhipp
66f0ff5b13
add ordering to search_by_attr that is used for model lists
2024-03-08 13:38:38 -06:00
maryhipp
281222df3c
remove old data migration from previous schema version
2024-03-08 13:10:27 -05:00
maryhipp
d5674150fa
ruff
2024-03-08 13:02:04 -05:00
maryhipp
0cb2cf6644
wrap version check in try/except
2024-03-08 13:02:04 -05:00
maryhipp
da87266c9c
remove log
2024-03-08 13:02:04 -05:00
maryhipp
35731a6f51
fix null description, add logging
2024-03-08 13:02:04 -05:00
Brandon Rising
a3dfa161a8
Run ruff
2024-03-08 13:02:04 -05:00
Brandon Rising
42d606f07c
use register instead of heuristic import, get rid of typing warnings
2024-03-08 13:02:04 -05:00
maryhipp
9063b1ae61
on model manager start, look to see if yaml needs to be migrated and do it if so
2024-03-08 13:02:04 -05:00
psychedelicious
7a87ebb3b2
fix(mm): add control adapter default settings to ModelRecordChanges schema
...
This is needed to update Control Adapter defaults.
2024-03-08 12:44:58 -05:00
psychedelicious
6fc3e86061
tidy(mm): only main models get the main default settings
2024-03-08 12:44:58 -05:00
psychedelicious
4a20377fef
tidy(config): move version "setting" to new CLIArgs category
...
It's not actually a setting.
2024-03-08 13:59:59 +11:00
psychedelicious
ed803640f7
tidy(mm): move remote_api_tokens to new ModelInstall category
2024-03-08 13:59:59 +11:00
psychedelicious
576bb4a61d
feat(mm): support generic API tokens via regex/token pairs in config
...
A list of regex and token pairs is accepted. As a file is downloaded by the model installer, the URL is tested against the provided regex/token pairs. The token for the first matching regex is used during download, added as a bearer token.
2024-03-08 13:59:59 +11:00
Brandon Rising
b6065d6328
Run ruff with newest version of ruff
2024-03-08 13:59:59 +11:00
Brandon Rising
04229f4a21
Run ruff
2024-03-08 13:59:59 +11:00
Brandon Rising
73a190fb6e
Add remote_repo_api_key config to be added as a token query param for all remote url model downloads
2024-03-08 13:59:59 +11:00
Brandon Rising
952d97741e
Remove civit ai from tests and documentation
2024-03-08 13:59:59 +11:00
Brandon Rising
d1f859a446
Remove civit AI model install resources
2024-03-08 13:59:59 +11:00
psychedelicious
5118160282
docs(mm): update comment about model images
2024-03-08 12:26:35 +11:00
psychedelicious
a3a42d25d3
fix(mm): model images reload when changed
...
When we change a model image, its URL remains the same. The browser will aggressively cache the image. The easiest way to fix this is to append a random query parameter to the URL whenever we build a model config in the API.
2024-03-08 12:26:35 +11:00
psychedelicious
9b48029bc9
tidy(mm): ModelImages service
2024-03-06 21:57:41 -05:00
Jennifer Player
4af5a09a68
cleanup
2024-03-06 21:57:41 -05:00
Jennifer Player
8df02623f2
cleanup
2024-03-06 21:57:41 -05:00
Jennifer Player
aa88fadc30
use webp images
2024-03-06 21:57:41 -05:00
Jennifer Player
8411029d93
get model image url from model config, added thumbnail formatting for images
2024-03-06 21:57:41 -05:00
Jennifer Player
239b1e8cc7
moved upload image field and added delete image functionality
2024-03-06 21:57:41 -05:00
Jennifer Player
2f6964bfa5
fetching model image, still not working
2024-03-06 21:57:41 -05:00
Jennifer Player
c1cdfd132b
moved model image to edit page, added model_images service
2024-03-06 21:57:41 -05:00
psychedelicious
528ac5dd25
refactor(nodes): model identifiers
...
- All models are identified by a key and optionally a submodel type via new model `ModelField`. Previously, a few model types had their own class, but not all of them. This inconsistency just added complexity without any benefit.
- Update all invocation to use the new format.
- In the node API, models are loaded by key or an instance of `ModelField` as a convenience.
- Add an enriched model schema for metadata. It includes key, hash, name, base and type.
2024-03-07 10:56:59 +11:00
psychedelicious
afd9ae7712
tidy(mm): remove convenience methods from high level model manager service
...
These were added as a hold-me-over for the nodes API changes, no longer needed. A followup commit will fix the nodes API to not rely on these.
2024-03-07 10:56:59 +11:00
maryhipp
cdc0d0c182
add config_path to ModelRecordChanges
2024-03-07 10:29:29 +11:00
psychedelicious
b0615bdfd4
fix(nodes): correctly serialize outputs
...
In order for delete by match to work, we need the whole invocation output to be stringified.
For some reason, the serialization of the output was set to only include the `type` field. It should instead include the whole output.
I don't understand how this ever worked unless pydantic had different serialization behaviour in v1 (though it appears to have been the same).
Closes #5805
2024-03-06 08:14:12 -05:00
psychedelicious
bab20467fb
fix(nodes): fix invocation cache clear method args
2024-03-06 08:14:12 -05:00
psychedelicious
e24624109e
fix(nodes): fix invocation cache ABC typing
2024-03-06 08:14:12 -05:00
psychedelicious
bcf58cac59
feat(mm): add config to skip model hash
...
This is useful for when you are using a memory DB and do not want to wait for all models to be hashed on startup.
2024-03-05 23:50:19 +11:00
psychedelicious
e8797787cf
fix(mm): fix incorrect calls to update_model
2024-03-05 23:50:19 +11:00
psychedelicious
0082ecb22b
feat(mm): add path to ModelRecordChanges
2024-03-05 23:50:19 +11:00
psychedelicious
656839fcd1
fix(mm): fix typing on heuristic_import
2024-03-05 23:50:19 +11:00
psychedelicious
78895b3e80
fix(mm): add missing inplace parameter to model install abc
2024-03-05 23:50:19 +11:00
psychedelicious
58fa9c2fac
fix(mm): do not allow extra fields on ModelRecordChanges
2024-03-05 23:50:19 +11:00
psychedelicious
5551cf8ac4
feat(mm): revise update_model to use ModelRecordChanges
2024-03-05 23:50:19 +11:00
psychedelicious
37b969d339
tidy(mm): add default_settings to model config
2024-03-05 23:50:19 +11:00
psychedelicious
c953e61294
tidy(mm): "trigger_words" -> "trigger_phrases"
2024-03-05 23:50:19 +11:00
psychedelicious
44c40d7d1a
refactor(mm): remove unused metadata logic, fix tests
...
- Metadata is merged with the config. We can simplify the MM substantially and remove the handling for metadata.
- Per discussion, we don't have an ETA for frontend implementation of tags, and with the realization that the tags from CivitAI are largely useless, there's no reason to keep tags in the MM right now. When we are ready to implement tags on the frontend, we can refer back to the implementation here and use it if it supports the design.
- Fix all tests.
2024-03-05 23:50:19 +11:00
psychedelicious
c3aa985c93
refactor(mm): get metadata working
2024-03-05 23:50:19 +11:00
psychedelicious
7cb0da1f66
refactor(mm): wip schema changes
2024-03-05 23:50:19 +11:00
psychedelicious
f13f5984c0
fix(mm): update db schema & migration
2024-03-05 23:50:19 +11:00
psychedelicious
9378e47a06
feat(mm): add source_type to model configs
2024-03-05 23:50:19 +11:00
psychedelicious
2c835fd550
refactor(mm): WIP db schema
2024-03-05 23:50:19 +11:00
psychedelicious
a8cd3dfc99
refactor(mm): add models table (schema WIP), rename "original_hash" -> "hash"
2024-03-05 23:50:19 +11:00
psychedelicious
9b40c28144
tidy(mm): rename ckpy "config" -> "config_path"
2024-03-05 23:50:19 +11:00
psychedelicious
76cbc745e1
refactor(mm): add CheckpointConfigBase for all ckpt models
2024-03-05 23:50:19 +11:00
psychedelicious
b880a31039
refactor(mm): remove ztsnr_training field on _MainConfig
...
This is used to determine the CFG Rescale Multiplier setting. We'll handle this in the UI as a default setting.
2024-03-05 23:50:19 +11:00
Mary Hipp Rogers
8b34f5298c
Default model settings ( #5850 )
...
* UI in MM to create trigger phrases
* add scheduler and vaePrecision to config
* UI for configuring default settings for models'
* hook MM default model settings up to API
* add button to set default settings in parameters
* pull out trigger phrases
* back-end for default settings
* lint
* remove log;
gi
* ruff
* ruff format
---------
Co-authored-by: Mary Hipp <maryhipp@Marys-MacBook-Air.local>
2024-03-04 09:39:03 -05:00
Brandon Rising
893bcd16fc
Next: Allow in place local installs of models
2024-03-04 23:11:41 +11:00
Ryan Dick
f6028a4c61
Log a stack trace for invocation errors.
2024-03-04 23:01:56 +11:00
psychedelicious
ae99428883
fix(mm): use UUIDv4 for key
...
This changes the functionality of this PR to only use the updated hashing for model hashes with a UUID for the key.
2024-03-03 14:32:14 +11:00
psychedelicious
86982f3059
feat(mm): make ModelHash instantiatable, taking an algorithm as arg
2024-03-03 14:32:14 +11:00
psychedelicious
982076d7d7
feat(mm): add hashing algos to ModelHash
...
- Some algos are slow, so it is now just called ModelHash
- Added all hashlib algos, plus BLAKE3 and the fast (but incorrect) SHA1 algo
2024-03-03 14:32:14 +11:00
Lincoln Stein
a72056e0df
make model key assignment deterministic
...
- When installing, model keys are now calculated from the model contents.
- .safetensors, .ckpt and other single file models are hashed with sha1
- The contents of diffusers directories are hashed using imohash (faster)
fixup yaml->sql db migration script to assign deterministic key
- this commit also detects and assigns the correct image encoder for
ip adapter models.
2024-03-03 14:32:14 +11:00
Lincoln Stein
0832e1818e
Fix problem of all installed models being assigned "<NOKEY>"
...
- Also fix redundant scanning of models directory at startup.
2024-02-29 22:31:05 -05:00
Lincoln Stein
813a086cfe
fix race condition between downloading last file and starting install
2024-03-01 10:42:33 +11:00
Lincoln Stein
e18533e3b5
add debugging statements and a timeout to download test
2024-03-01 10:42:33 +11:00
psychedelicious
dd9daf8efb
chore: ruff
2024-03-01 10:42:33 +11:00
psychedelicious
753919c6d7
docs(nodes): update all docstrings for public nodes API
2024-03-01 10:42:33 +11:00
psychedelicious
0b0128647b
feat(nodes): revise model load API args
2024-03-01 10:42:33 +11:00
psychedelicious
12e859835b
feat(mm): add log stmt for download complete event
2024-03-01 10:42:33 +11:00
psychedelicious
ea7b7bcf40
chore: ruff
2024-03-01 10:42:33 +11:00
psychedelicious
e22c4987bf
chore: ruff
2024-03-01 10:42:33 +11:00
maryhipp
26a209a00d
add error_reason to ModelInstallJob
2024-03-01 10:42:33 +11:00
Jennifer Player
ea364bdf82
delete model imports and prune all finished, update state with socket messages
2024-03-01 10:42:33 +11:00
Brandon Rising
c778ab8db4
Allow passing in key on register
2024-03-01 10:42:33 +11:00
Brandon Rising
65b91356d0
Remove passing keys in on register
2024-03-01 10:42:33 +11:00
Brandon Rising
de9287a3e4
Run ruff
2024-03-01 10:42:33 +11:00
Brandon Rising
008716040b
Allow users to run model manager without cuda
2024-03-01 10:42:33 +11:00
Lincoln Stein
cc41e8912c
several small model install enhancements
...
- Support extended HF repoid syntax in TUI. This allows
installation of subfolders and safetensors files, as in
`XpucT/Deliberate::Deliberate_v5.safetensors`
- Add `error` and `error_traceback` properties to the install
job objects.
- Rename the `heuristic_import` route to `heuristic_install`.
- Fix the example `config` input in the `heuristic_install` route.
2024-03-01 10:42:33 +11:00
Lincoln Stein
65dd4f4abc
fix repo-id for the Deliberate v5 model
...
prevent lora and embedding file suffixes from being stripped during installation
apply psychedelicious patch to get compel to load proper TI embedding
2024-03-01 10:42:33 +11:00
Lincoln Stein
5bb3aeaccd
remove startup dependency on legacy models.yaml file
2024-03-01 10:42:33 +11:00
psychedelicious
a37b60db13
feat(bulk_download): update response model, messages
2024-03-01 10:42:33 +11:00
psychedelicious
cbb997e7d0
tidy(bulk_download): don't store events service separately
...
Using the invoker object directly leaves no ambiguity as to what `_events_bus` actually is.
2024-03-01 10:42:33 +11:00
psychedelicious
98441ad08d
tidy(bulk_download): do not rely on pagination API to get all images for board
...
We can get all images for the board as a list of image names, then pass that to `_image_handler` to get the DTOs, decoupling from the pagination API.
2024-03-01 10:42:33 +11:00
psychedelicious
80c67dd6e0
tidy(bulk_download): nit - use or as a coalescing operator
...
Just a bit cleaner.
2024-03-01 10:42:33 +11:00
psychedelicious
38af234108
tidy(bulk_download): use single underscore for private attrs
...
Double underscores are used in the app but it doesn't actually do or convey anything that single underscores don't already do. Considered unpythonic except for actual dunder/magic methods.
2024-03-01 10:42:33 +11:00
psychedelicious
2291122c2b
tidy(bulk_download): remove class-level attr annotations
...
These can be misleading as they shadow actual assigned class attributes. This pattern is in the rest of the app but it shouldn't be.
2024-03-01 10:42:33 +11:00
psychedelicious
bf3b10cb1c
tidy(bulk_download): remove extraneous abstract methods
...
`start`, `stop` and `__init__` are not required in implementations of an ABC or service.
2024-03-01 10:42:33 +11:00
psychedelicious
7f8f182a00
tidy(bulk_download): clean up comments
2024-03-01 10:42:33 +11:00
Stefan Tobler
e51867756a
adding bulk_download_item_name to socket events
2024-03-01 10:42:33 +11:00
Stefan Tobler
a8d7cf4e97
refactoring handlers to do null check
2024-03-01 10:42:33 +11:00
Stefan Tobler
037cac8154
removing dependency on an output folder, embrace python temp folder for bulk download
2024-03-01 10:42:33 +11:00
Stefan Tobler
b5a9ed351d
moving the responsibility of cleaning up board names to the service not the route
2024-03-01 10:42:33 +11:00
Stefan Tobler
f15aa562c2
using temp directory for downloads
2024-03-01 10:42:33 +11:00
Stefan Tobler
d0f3571e59
returning the bulk_download_item_name on response for possible polling
2024-03-01 10:42:33 +11:00
Stefan Tobler
b5ca1643a6
narrowing bulk_download stop service scope
2024-03-01 10:42:33 +11:00
Stefan Tobler
79eb871683
cleaning up bulk download zip after the response is complete
2024-03-01 10:42:33 +11:00
Stefan Tobler
7d91426d8f
refactoring bulk_download to be better managed
2024-03-01 10:42:33 +11:00
Stefan Tobler
795fbf0e81
refactoring bulkdownload to consider image category
2024-03-01 10:42:33 +11:00
Stefan Tobler
7114d64b86
fixing issue where default board did not return images
2024-03-01 10:42:33 +11:00
Stefan Tobler
c43ea9f25c
using the board name to download boards
2024-03-01 10:42:33 +11:00
Stefan Tobler
52b0deb179
reworking some of the logic to use a default room, adding endpoint to download file on complete
2024-03-01 10:42:33 +11:00
Stefan Tobler
7ecc18938b
linted and styling
2024-03-01 10:42:33 +11:00
Stefan Tobler
56d2d220a8
implementation of bulkdownload background task
2024-03-01 10:42:33 +11:00
Stefan Tobler
f1967c3393
adding socket events for bulk download
2024-03-01 10:42:33 +11:00
Stefan Tobler
812e24cbd2
groundwork for the bulk_download_service
2024-03-01 10:42:33 +11:00
psychedelicious
89fa36a818
chore(nodes): update TODO comment
2024-03-01 10:42:33 +11:00
psychedelicious
e3f9da29ba
tidy(nodes): clean up profiler/stats in processor, better comments
2024-03-01 10:42:33 +11:00
psychedelicious
763debdeeb
fix(nodes): fix typing on stats service context manager
2024-03-01 10:42:33 +11:00
psychedelicious
8bf9fd34ad
fix(nodes): fix model load events
...
was accessing incorrect properties in event data
2024-03-01 10:42:33 +11:00
psychedelicious
0b0cb0ccc6
feat(nodes): making invocation class var in processor
2024-03-01 10:42:33 +11:00
psychedelicious
fa39523b11
feat(nodes): improved error messages in processor
2024-03-01 10:42:33 +11:00
psychedelicious
16676feea8
feat(nodes): make processor thread limit and polling interval configurable
2024-03-01 10:42:33 +11:00
psychedelicious
d53a2a2d4e
chore(nodes): better comments for invocation context
2024-03-01 10:42:33 +11:00
psychedelicious
ccfe6b6bef
chore(nodes): "context_data" -> "data"
...
Changed within InvocationContext, for brevity.
2024-03-01 10:42:33 +11:00
psychedelicious
fdac0c3c9b
refactor(nodes): move is_canceled to context.util
2024-03-01 10:42:33 +11:00
psychedelicious
18adcc1dd2
feat(nodes): add whole queue_item to InvocationContextData
...
No reason to not have the whole thing in there.
2024-03-01 10:42:33 +11:00
psychedelicious
86c50f2d5b
tidy(nodes): remove extraneous comments
2024-03-01 10:42:33 +11:00
psychedelicious
3cfac8b843
feat(nodes): better invocation error messages
2024-03-01 10:42:33 +11:00
psychedelicious
0788b6ecee
chore(nodes): add comments for cancel state
2024-03-01 10:42:33 +11:00
psychedelicious
317d076a1a
feat(nodes): promote is_canceled to public node API
2024-03-01 10:42:33 +11:00
