Commit Graph

304 Commits

Author SHA1 Message Date
5ecfa86cd0 tests: fix test on macos 2024-03-19 09:24:28 +11:00
f69938c6a8 fix(config): revised config methods
- `write_file` requires an destination file path
- `read_config` -> `merge_from_file`, if no path is provided, reads from `self.init_file_path`
- update app, tests to use new methods
- fix configurator, was overwriting config file data unexpectedly
2024-03-19 09:24:28 +11:00
fbbf9c01b5 tests: fix remaining tests 2024-03-19 09:24:28 +11:00
15cef98a8b tests: fix docs test 2024-03-19 09:24:28 +11:00
5606f4d627 tests: redo config tests 2024-03-19 09:24:28 +11:00
e3f29ed320 tests: update default settings tests 2024-03-14 16:03:37 +11:00
9fcd67b5c0 feat(mm): add algorithm prefix to hashes
For example:
- md5:a0cd925fc063f98dbf029eee315060c3
- sha1:9e362940e5603fdc60566ea100a288ba2fe48b8c
- blake3:ce3f0c5f3c05d119f4a5dcaf209b50d3149046a0d3a9adee9fed4c83cad6b4d0
2024-03-14 15:54:42 +11:00
eb6e6548ed feat(mm): faster hashing for spinning disk HDDs
BLAKE3 has poor performance on spinning disks when parallelized. See https://github.com/BLAKE3-team/BLAKE3/issues/31

- Replace `skip_model_hash` setting with `hashing_algorithm`. Any algorithm we support is accepted.
- Add `random` algorithm: hashes a UUID with BLAKE3 to create a random "hash". Equivalent to the previous skip functionality.
- Add `blake3_single` algorithm: hashes on a single thread using BLAKE3, fixes the aforementioned performance issue
- Update model probe to accept the algorithm to hash with as an optional arg, defaulting to `blake3`
- Update all calls of the probe to use the app's configured hashing algorithm
- Update an external script that probes models
- Update tests
- Move ModelHash into its own module to avoid circuclar import issues
2024-03-14 15:54:42 +11:00
bd5b43c00d tidy(mm): ModelSearch cleanup
- No need for it to by a pydantic model. Just a class now.
- Remove ABC, it made it hard to understand what was going on as attributes were spread across the ABC and implementation. Also, there is no other implementation.
- Add tests
2024-03-10 12:09:47 +11:00
0696094d95 tests: fix tests
The tests were testing deprecated settings (not the settings themselves, just the class's functionality).
2024-03-08 16:31:39 -05:00
d156fd2093 tests: validate config docstring is current 2024-03-08 16:31:39 -05:00
df12e12e09 Run ruff 2024-03-08 15:36:14 -05:00
ee38fbe89c Remove check for models dir in model deletion, update tests to always assume the model path is an absolute path 2024-03-08 15:36:14 -05:00
2410ed689a tests(mm): add tests for control adapter probe default settings 2024-03-08 12:44:58 -05:00
952d97741e Remove civit ai from tests and documentation 2024-03-08 13:59:59 +11:00
347f1fd0b7 fix tests 2024-03-06 21:57:41 -05:00
afd9ae7712 tidy(mm): remove convenience methods from high level model manager service
These were added as a hold-me-over for the nodes API changes, no longer needed. A followup commit will fix the nodes API to not rely on these.
2024-03-07 10:56:59 +11:00
d3ab08fe10 tests: add invocation cache tests 2024-03-06 08:14:12 -05:00
7c9128b253 tidy(mm): use canonical capitalization for all model-related enums, classes
For example, "Lora" -> "LoRA", "Vae" -> "VAE".
2024-03-05 23:50:19 +11:00
6829610a71 tests: rename "example_config" -> "example_it_config" 2024-03-05 23:50:19 +11:00
5551cf8ac4 feat(mm): revise update_model to use ModelRecordChanges 2024-03-05 23:50:19 +11:00
02bde7bb75 tests: fix test_hf_model_select::test_select_multiple_weights on windows 2024-03-05 23:50:19 +11:00
44c40d7d1a refactor(mm): remove unused metadata logic, fix tests
- Metadata is merged with the config. We can simplify the MM substantially and remove the handling for metadata.
- Per discussion, we don't have an ETA for frontend implementation of tags, and with the realization that the tags from CivitAI are largely useless, there's no reason to keep tags in the MM right now. When we are ready to implement tags on the frontend, we can refer back to the implementation here and use it if it supports the design.
- Fix all tests.
2024-03-05 23:50:19 +11:00
3534366146 fix(mm): fix extraneous downloaded files in diffusers
Sometimes, diffusers model components (tokenizer, unet, etc.) have multiple weights files in the same directory.

In this situation, we assume the files are different versions of the same weights. For example, we may have multiple
formats (`.bin`, `.safetensors`) with different precisions. When downloading model files, we want to select only
the best of these files for the requested format and precision/variant.

The previous logic assumed that each model weights file would have the same base filename, but this assumption was
not always true. The logic is revised score each file and choose the best scoring file, resulting in only a single
file being downloaded for each submodel/subdirectory.
2024-03-05 23:50:19 +11:00
a8cd3dfc99 refactor(mm): add models table (schema WIP), rename "original_hash" -> "hash" 2024-03-05 23:50:19 +11:00
0cce582f2f tidy(mm): remove current_hash 2024-03-05 23:50:19 +11:00
4347d1c7f7 tests(mm): fix some objects in tests 2024-03-05 23:50:19 +11:00
9b40c28144 tidy(mm): rename ckpy "config" -> "config_path" 2024-03-05 23:50:19 +11:00
2f372d9b18 tests(mm): update tests to reflect using UUID for key 2024-03-03 14:32:14 +11:00
554d175792 feat(mm): improved model hash class
- Use memory view for hashlib algorithms (closer to python 3.11's filehash API in hashlib)
- Remove `sha1_fast` (realized it doesn't even hash the whole file, it just does the first block)
- Add support for custom file filters
- Update docstrings
- Update tests
2024-03-03 14:32:14 +11:00
863ce00712 tests(mm): add tests for ModelHash 2024-03-03 14:32:14 +11:00
8fb297e5f6 add regression tests for <NOKEY> issue 2024-02-29 22:31:05 -05:00
3a2afe1d15 chore: ruff 2024-03-01 10:42:33 +11:00
813a086cfe fix race condition between downloading last file and starting install 2024-03-01 10:42:33 +11:00
e18533e3b5 add debugging statements and a timeout to download test 2024-03-01 10:42:33 +11:00
dd9daf8efb chore: ruff 2024-03-01 10:42:33 +11:00
94e3857110 handle change to Civitai metadata schema for commercial usage 2024-03-01 10:42:33 +11:00
cc41e8912c several small model install enhancements
- Support extended HF repoid syntax in TUI. This allows
  installation of subfolders and safetensors files, as in
  `XpucT/Deliberate::Deliberate_v5.safetensors`

- Add `error` and `error_traceback` properties to the install
  job objects.

- Rename the `heuristic_import` route to `heuristic_install`.

- Fix the example `config` input in the `heuristic_install` route.
2024-03-01 10:42:33 +11:00
5cba55d670 test: clean up & fix tests
- Deduplicate the mock invocation services. This is possible now that the import order issue is resolved.
- Merge `DummyEventService` into `TestEventService` and update all tests to use `TestEventService`.
2024-03-01 10:42:33 +11:00
037cac8154 removing dependency on an output folder, embrace python temp folder for bulk download 2024-03-01 10:42:33 +11:00
0ab9fe6987 relocating event_service fixture due to import ordering 2024-03-01 10:42:33 +11:00
b5a9ed351d moving the responsibility of cleaning up board names to the service not the route 2024-03-01 10:42:33 +11:00
5f4b406cfe updating imports to satisfy ruff 2024-03-01 10:42:33 +11:00
f15aa562c2 using temp directory for downloads 2024-03-01 10:42:33 +11:00
d0f3571e59 returning the bulk_download_item_name on response for possible polling 2024-03-01 10:42:33 +11:00
b5ca1643a6 narrowing bulk_download stop service scope 2024-03-01 10:42:33 +11:00
39c01a833d adding test coverage for new bulk download routes 2024-03-01 10:42:33 +11:00
79eb871683 cleaning up bulk download zip after the response is complete 2024-03-01 10:42:33 +11:00
7544b350f3 replacing import removed during rebase 2024-03-01 10:42:33 +11:00
284ba041bd 97% test coverage on bulk_download 2024-03-01 10:42:33 +11:00