Commit Graph

304 Commits

Author SHA1 Message Date
psychedelicious
5ecfa86cd0 tests: fix test on macos 2024-03-19 09:24:28 +11:00
psychedelicious
f69938c6a8 fix(config): revised config methods
- `write_file` requires an destination file path
- `read_config` -> `merge_from_file`, if no path is provided, reads from `self.init_file_path`
- update app, tests to use new methods
- fix configurator, was overwriting config file data unexpectedly
2024-03-19 09:24:28 +11:00
psychedelicious
fbbf9c01b5 tests: fix remaining tests 2024-03-19 09:24:28 +11:00
psychedelicious
15cef98a8b tests: fix docs test 2024-03-19 09:24:28 +11:00
psychedelicious
5606f4d627 tests: redo config tests 2024-03-19 09:24:28 +11:00
psychedelicious
e3f29ed320 tests: update default settings tests 2024-03-14 16:03:37 +11:00
psychedelicious
9fcd67b5c0 feat(mm): add algorithm prefix to hashes
For example:
- md5:a0cd925fc063f98dbf029eee315060c3
- sha1:9e362940e5603fdc60566ea100a288ba2fe48b8c
- blake3:ce3f0c5f3c05d119f4a5dcaf209b50d3149046a0d3a9adee9fed4c83cad6b4d0
2024-03-14 15:54:42 +11:00
psychedelicious
eb6e6548ed feat(mm): faster hashing for spinning disk HDDs
BLAKE3 has poor performance on spinning disks when parallelized. See https://github.com/BLAKE3-team/BLAKE3/issues/31

- Replace `skip_model_hash` setting with `hashing_algorithm`. Any algorithm we support is accepted.
- Add `random` algorithm: hashes a UUID with BLAKE3 to create a random "hash". Equivalent to the previous skip functionality.
- Add `blake3_single` algorithm: hashes on a single thread using BLAKE3, fixes the aforementioned performance issue
- Update model probe to accept the algorithm to hash with as an optional arg, defaulting to `blake3`
- Update all calls of the probe to use the app's configured hashing algorithm
- Update an external script that probes models
- Update tests
- Move ModelHash into its own module to avoid circuclar import issues
2024-03-14 15:54:42 +11:00
psychedelicious
bd5b43c00d tidy(mm): ModelSearch cleanup
- No need for it to by a pydantic model. Just a class now.
- Remove ABC, it made it hard to understand what was going on as attributes were spread across the ABC and implementation. Also, there is no other implementation.
- Add tests
2024-03-10 12:09:47 +11:00
psychedelicious
0696094d95 tests: fix tests
The tests were testing deprecated settings (not the settings themselves, just the class's functionality).
2024-03-08 16:31:39 -05:00
psychedelicious
d156fd2093 tests: validate config docstring is current 2024-03-08 16:31:39 -05:00
Brandon Rising
df12e12e09 Run ruff 2024-03-08 15:36:14 -05:00
Brandon Rising
ee38fbe89c Remove check for models dir in model deletion, update tests to always assume the model path is an absolute path 2024-03-08 15:36:14 -05:00
psychedelicious
2410ed689a tests(mm): add tests for control adapter probe default settings 2024-03-08 12:44:58 -05:00
Brandon Rising
952d97741e Remove civit ai from tests and documentation 2024-03-08 13:59:59 +11:00
Jennifer Player
347f1fd0b7 fix tests 2024-03-06 21:57:41 -05:00
psychedelicious
afd9ae7712 tidy(mm): remove convenience methods from high level model manager service
These were added as a hold-me-over for the nodes API changes, no longer needed. A followup commit will fix the nodes API to not rely on these.
2024-03-07 10:56:59 +11:00
psychedelicious
d3ab08fe10 tests: add invocation cache tests 2024-03-06 08:14:12 -05:00
psychedelicious
7c9128b253 tidy(mm): use canonical capitalization for all model-related enums, classes
For example, "Lora" -> "LoRA", "Vae" -> "VAE".
2024-03-05 23:50:19 +11:00
psychedelicious
6829610a71 tests: rename "example_config" -> "example_it_config" 2024-03-05 23:50:19 +11:00
psychedelicious
5551cf8ac4 feat(mm): revise update_model to use ModelRecordChanges 2024-03-05 23:50:19 +11:00
psychedelicious
02bde7bb75 tests: fix test_hf_model_select::test_select_multiple_weights on windows 2024-03-05 23:50:19 +11:00
psychedelicious
44c40d7d1a refactor(mm): remove unused metadata logic, fix tests
- Metadata is merged with the config. We can simplify the MM substantially and remove the handling for metadata.
- Per discussion, we don't have an ETA for frontend implementation of tags, and with the realization that the tags from CivitAI are largely useless, there's no reason to keep tags in the MM right now. When we are ready to implement tags on the frontend, we can refer back to the implementation here and use it if it supports the design.
- Fix all tests.
2024-03-05 23:50:19 +11:00
psychedelicious
3534366146 fix(mm): fix extraneous downloaded files in diffusers
Sometimes, diffusers model components (tokenizer, unet, etc.) have multiple weights files in the same directory.

In this situation, we assume the files are different versions of the same weights. For example, we may have multiple
formats (`.bin`, `.safetensors`) with different precisions. When downloading model files, we want to select only
the best of these files for the requested format and precision/variant.

The previous logic assumed that each model weights file would have the same base filename, but this assumption was
not always true. The logic is revised score each file and choose the best scoring file, resulting in only a single
file being downloaded for each submodel/subdirectory.
2024-03-05 23:50:19 +11:00
psychedelicious
a8cd3dfc99 refactor(mm): add models table (schema WIP), rename "original_hash" -> "hash" 2024-03-05 23:50:19 +11:00
psychedelicious
0cce582f2f tidy(mm): remove current_hash 2024-03-05 23:50:19 +11:00
psychedelicious
4347d1c7f7 tests(mm): fix some objects in tests 2024-03-05 23:50:19 +11:00
psychedelicious
9b40c28144 tidy(mm): rename ckpy "config" -> "config_path" 2024-03-05 23:50:19 +11:00
psychedelicious
2f372d9b18 tests(mm): update tests to reflect using UUID for key 2024-03-03 14:32:14 +11:00
psychedelicious
554d175792 feat(mm): improved model hash class
- Use memory view for hashlib algorithms (closer to python 3.11's filehash API in hashlib)
- Remove `sha1_fast` (realized it doesn't even hash the whole file, it just does the first block)
- Add support for custom file filters
- Update docstrings
- Update tests
2024-03-03 14:32:14 +11:00
psychedelicious
863ce00712 tests(mm): add tests for ModelHash 2024-03-03 14:32:14 +11:00
Lincoln Stein
8fb297e5f6 add regression tests for <NOKEY> issue 2024-02-29 22:31:05 -05:00
psychedelicious
3a2afe1d15 chore: ruff 2024-03-01 10:42:33 +11:00
Lincoln Stein
813a086cfe fix race condition between downloading last file and starting install 2024-03-01 10:42:33 +11:00
Lincoln Stein
e18533e3b5 add debugging statements and a timeout to download test 2024-03-01 10:42:33 +11:00
psychedelicious
dd9daf8efb chore: ruff 2024-03-01 10:42:33 +11:00
Lincoln Stein
94e3857110 handle change to Civitai metadata schema for commercial usage 2024-03-01 10:42:33 +11:00
Lincoln Stein
cc41e8912c several small model install enhancements
- Support extended HF repoid syntax in TUI. This allows
  installation of subfolders and safetensors files, as in
  `XpucT/Deliberate::Deliberate_v5.safetensors`

- Add `error` and `error_traceback` properties to the install
  job objects.

- Rename the `heuristic_import` route to `heuristic_install`.

- Fix the example `config` input in the `heuristic_install` route.
2024-03-01 10:42:33 +11:00
psychedelicious
5cba55d670 test: clean up & fix tests
- Deduplicate the mock invocation services. This is possible now that the import order issue is resolved.
- Merge `DummyEventService` into `TestEventService` and update all tests to use `TestEventService`.
2024-03-01 10:42:33 +11:00
Stefan Tobler
037cac8154 removing dependency on an output folder, embrace python temp folder for bulk download 2024-03-01 10:42:33 +11:00
Stefan Tobler
0ab9fe6987 relocating event_service fixture due to import ordering 2024-03-01 10:42:33 +11:00
Stefan Tobler
b5a9ed351d moving the responsibility of cleaning up board names to the service not the route 2024-03-01 10:42:33 +11:00
Stefan Tobler
5f4b406cfe updating imports to satisfy ruff 2024-03-01 10:42:33 +11:00
Stefan Tobler
f15aa562c2 using temp directory for downloads 2024-03-01 10:42:33 +11:00
Stefan Tobler
d0f3571e59 returning the bulk_download_item_name on response for possible polling 2024-03-01 10:42:33 +11:00
Stefan Tobler
b5ca1643a6 narrowing bulk_download stop service scope 2024-03-01 10:42:33 +11:00
Stefan Tobler
39c01a833d adding test coverage for new bulk download routes 2024-03-01 10:42:33 +11:00
Stefan Tobler
79eb871683 cleaning up bulk download zip after the response is complete 2024-03-01 10:42:33 +11:00
Stefan Tobler
7544b350f3 replacing import removed during rebase 2024-03-01 10:42:33 +11:00
Stefan Tobler
284ba041bd 97% test coverage on bulk_download 2024-03-01 10:42:33 +11:00