Remove civit ai from tests and documentation

docs/contributing/MODEL_MANAGER.md

@@ -16,11 +16,6 @@ model. These are the:
   information. It is also responsible for managing the InvokeAI
   `models` directory and its contents.
   
-* _ModelMetadataStore_ and _ModelMetaDataFetch_ Backend modules that
-  are able to retrieve metadata from online model repositories,
-  transform them into Pydantic models, and cache them to the InvokeAI
-  SQL database.
-  
 * _DownloadQueueServiceBase_
   A multithreaded downloader responsible
   for downloading models from a remote source to disk. The download
@@ -382,17 +377,14 @@ functionality:
   
 * Downloading a model from an arbitrary URL and installing it in
   `models_dir`.
-  
+
-* Special handling for Civitai model URLs which allow the user to
-  paste in a model page's URL or download link
-  
 * Special handling for HuggingFace repo_ids to recursively download
   the contents of the repository, paying attention to alternative
   variants such as fp16.
   
 * Saving tags and other metadata about the model into the invokeai database
   when fetching from a repo that provides that type of information,
-  (currently only Civitai and HuggingFace).
+  (currently only HuggingFace).
   
 ### Initializing the installer
 
@@ -436,7 +428,6 @@ required parameters:
 | `app_config`         | InvokeAIAppConfig       | InvokeAI app configuration object |
 | `record_store`   | ModelRecordServiceBase  | Config record storage database |
 | `download_queue`   | DownloadQueueServiceBase  | Download queue object |
-| `metadata_store`   | Optional[ModelMetadataStore]  | Metadata storage object |
 |`session`           | Optional[requests.Session]    | Swap in a different Session object (usually for debugging) |
 
 Once initialized, the installer will provide the following methods:
@@ -580,8 +571,7 @@ The `AnyHttpUrl` class can be imported from `pydantic.networks`.
 
 Ordinarily, no metadata is retrieved from these sources. However,
 there is special-case code in the installer that looks for HuggingFace
-and Civitai URLs and fetches the corresponding model metadata from
+and fetches the corresponding model metadata from the corresponding repo.
-the corresponding repo.
 
 #### HFModelSource
 
@@ -1228,9 +1218,9 @@ queue and have not yet reached a terminal state.
 
 The modules found under `invokeai.backend.model_manager.metadata`
 provide a straightforward API for fetching model metadatda from online
-repositories. Currently two repositories are supported: HuggingFace
+repositories. Currently only HuggingFace is supported. However, the
-and Civitai. However, the modules are easily extended for additional
+modules are easily extended for additional repos, provided that they
-repos, provided that they have defined APIs for metadata access.
+have defined APIs for metadata access.
 
 Metadata comprises any descriptive information that is not essential
 for getting the model to run. For example "author" is metadata, while
@@ -1242,37 +1232,16 @@ model's config, as defined in `invokeai.backend.model_manager.config`.
 ```
 from invokeai.backend.model_manager.metadata import (
    AnyModelRepoMetadata,
-   CivitaiMetadataFetch,
-   CivitaiMetadata
-   ModelMetadataStore,
 )
 # to access the initialized sql database
 from invokeai.app.api.dependencies import ApiDependencies
 
-civitai = CivitaiMetadataFetch()
+hf = HuggingFaceMetadataFetch()
 
 # fetch the metadata
-model_metadata = civitai.from_url("https://civitai.com/models/215796")
+model_metadata = hf.from_id("<repo_id>")
 
-# get some common metadata fields
+assert isinstance(model_metadata, HuggingFaceMetadata)
-author = model_metadata.author
-tags = model_metadata.tags
-
-# get some Civitai-specific fields
-assert isinstance(model_metadata, CivitaiMetadata)
-
-trained_words = model_metadata.trained_words
-base_model = model_metadata.base_model_trained_on
-thumbnail = model_metadata.thumbnail_url
-
-# cache the metadata to the database using the key corresponding to
-# an existing model config record in the `model_config` table
-sql_cache = ModelMetadataStore(ApiDependencies.invoker.services.db)
-sql_cache.add_metadata('fb237ace520b6716adc98bcb16e8462c', model_metadata)
-
-# now we can search the database by tag, author or model name
-# matches will contain a list of model keys that match the search
-matches = sql_cache.search_by_tag({"tool", "turbo"})
 ```
 
 ### Structure of the Metadata objects
@@ -1309,52 +1278,14 @@ This descends from `ModelMetadataBase` and adds the following fields:
 | `last_modified`| datetime         | Date of last commit of this model to the repo |
 | `files`        | List[Path]       | List of the files in the model repo |
 
-#### `CivitaiMetadata`
-
-This descends from `ModelMetadataBase` and adds the following fields:
-
-| **Field Name** | **Type**        |  **Description** |
-|----------------|-----------------|------------------|
-| `type`         | Literal["civitai"]   | Used for the discriminated union of metadata classes|
-| `id`           | int                  | Civitai model id |
-| `version_name` | str                  | Name of this version of the model (distinct from model name) |
-| `version_id`   | int                  | Civitai model version id (distinct from model id) |
-| `created`      | datetime             | Date this version of the model was created |
-| `updated`      | datetime             | Date this version of the model was last updated |
-| `published`    | datetime             | Date this version of the model was published to Civitai |
-| `description`  | str                  | Model description. Quite verbose and contains HTML tags |
-| `version_description` | str           | Model version description, usually describes changes to the model |
-| `nsfw`         | bool                 | Whether the model tends to generate NSFW content |
-| `restrictions` | LicenseRestrictions  | An object that describes what is and isn't allowed with this model |
-| `trained_words`| Set[str]             | Trigger words for this model, if any |
-| `download_url` | AnyHttpUrl           | URL for downloading this version of the model |
-| `base_model_trained_on` | str         | Name of the model that this version was trained on |
-| `thumbnail_url` | AnyHttpUrl          | URL to access a representative thumbnail image of the model's output |
-| `weight_min`    | int                 | For LoRA sliders, the minimum suggested weight to apply |
-| `weight_max`    | int                 | For LoRA sliders, the maximum suggested weight to apply |
-
-Note that `weight_min` and `weight_max` are not currently populated
-and take the default values of (-1.0, +2.0). The issue is that these
-values aren't part of the structured data but appear in the text
-description. Some regular expression or LLM coding may be able to
-extract these values.
-
-Also be aware that `base_model_trained_on` is free text and doesn't
-correspond to our `ModelType` enum.
-
-`CivitaiMetadata` also defines some convenience properties relating to
-licensing restrictions: `credit_required`, `allow_commercial_use`,
-`allow_derivatives` and `allow_different_license`.
-
 #### `AnyModelRepoMetadata`
 
-This is a discriminated Union of `CivitaiMetadata` and
+This is a discriminated Union of `HuggingFaceMetadata`.
-`HuggingFaceMetadata`.
 
 ### Fetching Metadata from Online Repos
 
-The `HuggingFaceMetadataFetch` and `CivitaiMetadataFetch` classes will
+The `HuggingFaceMetadataFetch` class will
-retrieve metadata from their corresponding repositories and return
+retrieve metadata from its corresponding repository and return
 `AnyModelRepoMetadata` objects. Their base class
 `ModelMetadataFetchBase` is an abstract class that defines two
 methods: `from_url()` and `from_id()`. The former accepts the type of
@@ -1372,96 +1303,17 @@ provide a `requests.Session` argument. This allows you to customize
 the low-level HTTP fetch requests and is used, for instance, in the
 testing suite to avoid hitting the internet.
 
-The HuggingFace and Civitai fetcher subclasses add additional
+The HuggingFace fetcher subclass add additional repo-specific fetching methods:
-repo-specific fetching methods:
 
 #### HuggingFaceMetadataFetch
 
 This overrides its base class `from_json()` method to return a
 `HuggingFaceMetadata` object directly.
 
-#### CivitaiMetadataFetch
-
-This adds the following methods:
-
-`from_civitai_modelid()` This takes the ID of a model, finds the
-default version of the model, and then retrieves the metadata for
-that version, returning a `CivitaiMetadata` object directly.
-
-`from_civitai_versionid()` This takes the ID of a model version and
-retrieves its metadata. Functionally equivalent to `from_id()`, the
-only difference is that it returna a `CivitaiMetadata` object rather
-than an `AnyModelRepoMetadata`.
-
 ### Metadata Storage
 
-The `ModelMetadataStore` provides a simple facility to store model
+The `ModelConfigBase` stores this response in the `source_api_response` field
-metadata in the `invokeai.db` database. The data is stored as a JSON
+as a JSON blob.
-blob, with a few common fields (`name`, `author`, `tags`) broken out
-to be searchable.
-
-When a metadata object is saved to the database, it is identified
-using the model key, _and this key must correspond to an existing
-model key in the model_config table_. There is a foreign key integrity
-constraint between the `model_config.id` field and the
-`model_metadata.id` field such that if you attempt to save metadata
-under an unknown key, the attempt will result in an
-`UnknownModelException`. Likewise, when a model is deleted from
-`model_config`, the deletion of the corresponding metadata record will
-be triggered.
-
-Tags are stored in a normalized fashion in the tables `model_tags` and
-`tags`. Triggers keep the tag table in sync with the `model_metadata`
-table.
-
-To create the storage object, initialize it with the InvokeAI
-`SqliteDatabase` object. This is often done this way:
-
-```
-from invokeai.app.api.dependencies import ApiDependencies
-metadata_store = ModelMetadataStore(ApiDependencies.invoker.services.db)
-```
-
-You can then access the storage with the following methods:
-
-#### `add_metadata(key, metadata)`
-
-Add the metadata using a previously-defined model key.
-
-There is currently no `delete_metadata()` method. The metadata will
-persist until the matching config is deleted from the `model_config`
-table.
-
-#### `get_metadata(key) -> AnyModelRepoMetadata`
-
-Retrieve the metadata corresponding to the model key.
-
-#### `update_metadata(key, new_metadata)`
-
-Update an existing metadata record with new metadata.
-
-#### `search_by_tag(tags: Set[str]) -> Set[str]`
-
-Given a set of tags, find models that are tagged with them. If
-multiple tags are provided then a matching model must be tagged with
-*all* the tags in the set. This method returns a set of model keys and
-is intended to be used in conjunction with the `ModelRecordService`:
-
-```
-model_config_store = ApiDependencies.invoker.services.model_records
-matches = metadata_store.search_by_tag({'license:other'})
-models = [model_config_store.get(x) for x in matches]
-```
-
-#### `search_by_name(name: str) -> Set[str]
-
-Find all model metadata records that have the given name and return a
-set of keys to the corresponding model config objects.
-
-#### `search_by_author(author: str) -> Set[str]
-
-Find all model metadata records that have the given author and return
-a set of keys to the corresponding model config objects.
 
 ***
 

invokeai/app/services/model_install/model_install_default.py

@@ -589,7 +589,7 @@ class ModelInstallService(ModelInstallServiceBase):
         )
 
     def _import_from_url(self, source: URLModelSource, config: Optional[Dict[str, Any]]) -> ModelInstallJob:
-        # URLs from Civitai or HuggingFace will be handled specially
+        # URLs from HuggingFace will be handled specially
         metadata = None
         fetcher = None
         try:

invokeai/backend/model_manager/metadata/__init__.py

@@ -10,12 +10,10 @@ from invokeai.backend.model_manager.metadata import(
    HuggingFaceMetadata,
 )
 
-from invokeai.backend.model_manager.metadata.fetch import CivitaiMetadataFetch
+from invokeai.backend.model_manager.metadata.fetch import HuggingFaceMetadataFetch
 
-data = CivitaiMetadataFetch().from_url("https://civitai.com/models/206883/split")
+data = HuggingFaceMetadataFetch().from_id("<REPO_ID>")
-assert isinstance(data, CivitaiMetadata)
+assert isinstance(data, HuggingFaceMetadata)
-if data.allow_commercial_use:
-   print("Commercial use of this model is allowed")
 """
 
 from .fetch import HuggingFaceMetadataFetch, ModelMetadataFetchBase

invokeai/backend/model_manager/metadata/fetch/__init__.py

@@ -3,15 +3,11 @@ Initialization file for invokeai.backend.model_manager.metadata.fetch
 
 Usage:
 from invokeai.backend.model_manager.metadata.fetch import (
-    CivitaiMetadataFetch,
     HuggingFaceMetadataFetch,
 )
-from invokeai.backend.model_manager.metadata import CivitaiMetadata
 
-data = CivitaiMetadataFetch().from_url("https://civitai.com/models/206883/split")
+data = HuggingFaceMetadataFetch().from_id("<repo_id>")
-assert isinstance(data, CivitaiMetadata)
+assert isinstance(data, HuggingFaceMetadata)
-if data.allow_commercial_use:
-   print("Commercial use of this model is allowed")
 """
 
 from .fetch_base import ModelMetadataFetchBase

invokeai/backend/model_manager/metadata/fetch/fetch_base.py

@@ -5,11 +5,10 @@ This module is the base class for subclasses that fetch metadata from model repo
 
 Usage:
 
-from invokeai.backend.model_manager.metadata.fetch import CivitAIMetadataFetch
+from invokeai.backend.model_manager.metadata.fetch import HuggingFaceMetadataFetch
 
-fetcher = CivitaiMetadataFetch()
+data = HuggingFaceMetadataFetch().from_id("<REPO_ID>")
-metadata = fetcher.from_url("https://civitai.com/models/206883/split")
+assert isinstance(data, HuggingFaceMetadata)
-print(metadata.trained_words)
 """
 
 from abc import ABC, abstractmethod

tests/backend/model_manager/model_manager_fixtures.py

@@ -184,7 +184,7 @@ def mm2_record_store(mm2_app_config: InvokeAIAppConfig) -> ModelRecordServiceBas
         variant=ModelVariantType.Normal,
         hash="111222333444",
         source="https://civitai.com/models/206883/split",
-        source_type=ModelSourceType.CivitAI,
+        source_type=ModelSourceType.Url,
     )
     config3 = MainDiffusersConfig(
         key="test_config_3",