mirror of
https://github.com/invoke-ai/InvokeAI
synced 2024-08-30 20:32:17 +00:00
c238a7f18b
Upgrade pydantic and fastapi to latest.
- pydantic~=2.4.2
- fastapi~=103.2
- fastapi-events~=0.9.1
**Big Changes**
There are a number of logic changes needed to support pydantic v2. Most changes are very simple, like using the new methods to serialized and deserialize models, but there are a few more complex changes.
**Invocations**
The biggest change relates to invocation creation, instantiation and validation.
Because pydantic v2 moves all validation logic into the rust pydantic-core, we may no longer directly stick our fingers into the validation pie.
Previously, we (ab)used models and fields to allow invocation fields to be optional at instantiation, but required when `invoke()` is called. We directly manipulated the fields and invocation models when calling `invoke()`.
With pydantic v2, this is much more involved. Changes to the python wrapper do not propagate down to the rust validation logic - you have to rebuild the model. This causes problem with concurrent access to the invocation classes and is not a free operation.
This logic has been totally refactored and we do not need to change the model any more. The details are in `baseinvocation.py`, in the `InputField` function and `BaseInvocation.invoke_internal()` method.
In the end, this implementation is cleaner.
**Invocation Fields**
In pydantic v2, you can no longer directly add or remove fields from a model.
Previously, we did this to add the `type` field to invocations.
**Invocation Decorators**
With pydantic v2, we instead use the imperative `create_model()` API to create a new model with the additional field. This is done in `baseinvocation.py` in the `invocation()` wrapper.
A similar technique is used for `invocation_output()`.
**Minor Changes**
There are a number of minor changes around the pydantic v2 models API.
**Protected `model_` Namespace**
All models' pydantic-provided methods and attributes are prefixed with `model_` and this is considered a protected namespace. This causes some conflict, because "model" means something to us, and we have a ton of pydantic models with attributes starting with "model_".
Forunately, there are no direct conflicts. However, in any pydantic model where we define an attribute or method that starts with "model_", we must tell set the protected namespaces to an empty tuple.
```py
class IPAdapterModelField(BaseModel):
model_name: str = Field(description="Name of the IP-Adapter model")
base_model: BaseModelType = Field(description="Base model")
model_config = ConfigDict(protected_namespaces=())
```
**Model Serialization**
Pydantic models no longer have `Model.dict()` or `Model.json()`.
Instead, we use `Model.model_dump()` or `Model.model_dump_json()`.
**Model Deserialization**
Pydantic models no longer have `Model.parse_obj()` or `Model.parse_raw()`, and there are no `parse_raw_as()` or `parse_obj_as()` functions.
Instead, you need to create a `TypeAdapter` object to parse python objects or JSON into a model.
```py
adapter_graph = TypeAdapter(Graph)
deserialized_graph_from_json = adapter_graph.validate_json(graph_json)
deserialized_graph_from_dict = adapter_graph.validate_python(graph_dict)
```
**Field Customisation**
Pydantic `Field`s no longer accept arbitrary args.
Now, you must put all additional arbitrary args in a `json_schema_extra` arg on the field.
**Schema Customisation**
FastAPI and pydantic schema generation now follows the OpenAPI version 3.1 spec.
This necessitates two changes:
- Our schema customization logic has been revised
- Schema parsing to build node templates has been revised
The specific aren't important, but this does present additional surface area for bugs.
**Performance Improvements**
Pydantic v2 is a full rewrite with a rust backend. This offers a substantial performance improvement (pydantic claims 5x to 50x depending on the task). We'll notice this the most during serialization and deserialization of sessions/graphs, which happens very very often - a couple times per node.
I haven't done any benchmarks, but anecdotally, graph execution is much faster. Also, very larges graphs - like with massive iterators - are much, much faster.
227 lines
10 KiB
Python
227 lines
10 KiB
Python
from typing import Optional
|
|
|
|
from pydantic import Field
|
|
|
|
from invokeai.app.invocations.baseinvocation import (
|
|
BaseInvocation,
|
|
BaseInvocationOutput,
|
|
InputField,
|
|
InvocationContext,
|
|
OutputField,
|
|
invocation,
|
|
invocation_output,
|
|
)
|
|
from invokeai.app.invocations.controlnet_image_processors import ControlField
|
|
from invokeai.app.invocations.ip_adapter import IPAdapterModelField
|
|
from invokeai.app.invocations.model import LoRAModelField, MainModelField, VAEModelField
|
|
from invokeai.app.invocations.primitives import ImageField
|
|
from invokeai.app.invocations.t2i_adapter import T2IAdapterField
|
|
from invokeai.app.util.model_exclude_null import BaseModelExcludeNull
|
|
|
|
from ...version import __version__
|
|
|
|
|
|
class LoRAMetadataField(BaseModelExcludeNull):
|
|
"""LoRA metadata for an image generated in InvokeAI."""
|
|
|
|
lora: LoRAModelField = Field(description="The LoRA model")
|
|
weight: float = Field(description="The weight of the LoRA model")
|
|
|
|
|
|
class IPAdapterMetadataField(BaseModelExcludeNull):
|
|
image: ImageField = Field(description="The IP-Adapter image prompt.")
|
|
ip_adapter_model: IPAdapterModelField = Field(description="The IP-Adapter model to use.")
|
|
weight: float = Field(description="The weight of the IP-Adapter model")
|
|
begin_step_percent: float = Field(
|
|
default=0, ge=0, le=1, description="When the IP-Adapter is first applied (% of total steps)"
|
|
)
|
|
end_step_percent: float = Field(
|
|
default=1, ge=0, le=1, description="When the IP-Adapter is last applied (% of total steps)"
|
|
)
|
|
|
|
|
|
class CoreMetadata(BaseModelExcludeNull):
|
|
"""Core generation metadata for an image generated in InvokeAI."""
|
|
|
|
app_version: str = Field(default=__version__, description="The version of InvokeAI used to generate this image")
|
|
generation_mode: Optional[str] = Field(
|
|
default=None,
|
|
description="The generation mode that output this image",
|
|
)
|
|
created_by: Optional[str] = Field(description="The name of the creator of the image")
|
|
positive_prompt: Optional[str] = Field(default=None, description="The positive prompt parameter")
|
|
negative_prompt: Optional[str] = Field(default=None, description="The negative prompt parameter")
|
|
width: Optional[int] = Field(default=None, description="The width parameter")
|
|
height: Optional[int] = Field(default=None, description="The height parameter")
|
|
seed: Optional[int] = Field(default=None, description="The seed used for noise generation")
|
|
rand_device: Optional[str] = Field(default=None, description="The device used for random number generation")
|
|
cfg_scale: Optional[float] = Field(default=None, description="The classifier-free guidance scale parameter")
|
|
steps: Optional[int] = Field(default=None, description="The number of steps used for inference")
|
|
scheduler: Optional[str] = Field(default=None, description="The scheduler used for inference")
|
|
clip_skip: Optional[int] = Field(
|
|
default=None,
|
|
description="The number of skipped CLIP layers",
|
|
)
|
|
model: Optional[MainModelField] = Field(default=None, description="The main model used for inference")
|
|
controlnets: Optional[list[ControlField]] = Field(default=None, description="The ControlNets used for inference")
|
|
ipAdapters: Optional[list[IPAdapterMetadataField]] = Field(
|
|
default=None, description="The IP Adapters used for inference"
|
|
)
|
|
t2iAdapters: Optional[list[T2IAdapterField]] = Field(default=None, description="The IP Adapters used for inference")
|
|
loras: Optional[list[LoRAMetadataField]] = Field(default=None, description="The LoRAs used for inference")
|
|
vae: Optional[VAEModelField] = Field(
|
|
default=None,
|
|
description="The VAE used for decoding, if the main model's default was not used",
|
|
)
|
|
|
|
# Latents-to-Latents
|
|
strength: Optional[float] = Field(
|
|
default=None,
|
|
description="The strength used for latents-to-latents",
|
|
)
|
|
init_image: Optional[str] = Field(default=None, description="The name of the initial image")
|
|
|
|
# SDXL
|
|
positive_style_prompt: Optional[str] = Field(default=None, description="The positive style prompt parameter")
|
|
negative_style_prompt: Optional[str] = Field(default=None, description="The negative style prompt parameter")
|
|
|
|
# SDXL Refiner
|
|
refiner_model: Optional[MainModelField] = Field(default=None, description="The SDXL Refiner model used")
|
|
refiner_cfg_scale: Optional[float] = Field(
|
|
default=None,
|
|
description="The classifier-free guidance scale parameter used for the refiner",
|
|
)
|
|
refiner_steps: Optional[int] = Field(default=None, description="The number of steps used for the refiner")
|
|
refiner_scheduler: Optional[str] = Field(default=None, description="The scheduler used for the refiner")
|
|
refiner_positive_aesthetic_score: Optional[float] = Field(
|
|
default=None, description="The aesthetic score used for the refiner"
|
|
)
|
|
refiner_negative_aesthetic_score: Optional[float] = Field(
|
|
default=None, description="The aesthetic score used for the refiner"
|
|
)
|
|
refiner_start: Optional[float] = Field(default=None, description="The start value used for refiner denoising")
|
|
|
|
|
|
class ImageMetadata(BaseModelExcludeNull):
|
|
"""An image's generation metadata"""
|
|
|
|
metadata: Optional[dict] = Field(
|
|
default=None,
|
|
description="The image's core metadata, if it was created in the Linear or Canvas UI",
|
|
)
|
|
graph: Optional[dict] = Field(default=None, description="The graph that created the image")
|
|
|
|
|
|
@invocation_output("metadata_accumulator_output")
|
|
class MetadataAccumulatorOutput(BaseInvocationOutput):
|
|
"""The output of the MetadataAccumulator node"""
|
|
|
|
metadata: CoreMetadata = OutputField(description="The core metadata for the image")
|
|
|
|
|
|
@invocation(
|
|
"metadata_accumulator", title="Metadata Accumulator", tags=["metadata"], category="metadata", version="1.0.0"
|
|
)
|
|
class MetadataAccumulatorInvocation(BaseInvocation):
|
|
"""Outputs a Core Metadata Object"""
|
|
|
|
generation_mode: Optional[str] = InputField(
|
|
default=None,
|
|
description="The generation mode that output this image",
|
|
)
|
|
positive_prompt: Optional[str] = InputField(default=None, description="The positive prompt parameter")
|
|
negative_prompt: Optional[str] = InputField(default=None, description="The negative prompt parameter")
|
|
width: Optional[int] = InputField(default=None, description="The width parameter")
|
|
height: Optional[int] = InputField(default=None, description="The height parameter")
|
|
seed: Optional[int] = InputField(default=None, description="The seed used for noise generation")
|
|
rand_device: Optional[str] = InputField(default=None, description="The device used for random number generation")
|
|
cfg_scale: Optional[float] = InputField(default=None, description="The classifier-free guidance scale parameter")
|
|
steps: Optional[int] = InputField(default=None, description="The number of steps used for inference")
|
|
scheduler: Optional[str] = InputField(default=None, description="The scheduler used for inference")
|
|
clip_skip: Optional[int] = InputField(
|
|
default=None,
|
|
description="The number of skipped CLIP layers",
|
|
)
|
|
model: Optional[MainModelField] = InputField(default=None, description="The main model used for inference")
|
|
controlnets: Optional[list[ControlField]] = InputField(
|
|
default=None, description="The ControlNets used for inference"
|
|
)
|
|
ipAdapters: Optional[list[IPAdapterMetadataField]] = InputField(
|
|
default=None, description="The IP Adapters used for inference"
|
|
)
|
|
t2iAdapters: Optional[list[T2IAdapterField]] = InputField(
|
|
default=None, description="The IP Adapters used for inference"
|
|
)
|
|
loras: Optional[list[LoRAMetadataField]] = InputField(default=None, description="The LoRAs used for inference")
|
|
strength: Optional[float] = InputField(
|
|
default=None,
|
|
description="The strength used for latents-to-latents",
|
|
)
|
|
init_image: Optional[str] = InputField(
|
|
default=None,
|
|
description="The name of the initial image",
|
|
)
|
|
vae: Optional[VAEModelField] = InputField(
|
|
default=None,
|
|
description="The VAE used for decoding, if the main model's default was not used",
|
|
)
|
|
|
|
# High resolution fix metadata.
|
|
hrf_width: Optional[int] = InputField(
|
|
default=None,
|
|
description="The high resolution fix height and width multipler.",
|
|
)
|
|
hrf_height: Optional[int] = InputField(
|
|
default=None,
|
|
description="The high resolution fix height and width multipler.",
|
|
)
|
|
hrf_strength: Optional[float] = InputField(
|
|
default=None,
|
|
description="The high resolution fix img2img strength used in the upscale pass.",
|
|
)
|
|
|
|
# SDXL
|
|
positive_style_prompt: Optional[str] = InputField(
|
|
default=None,
|
|
description="The positive style prompt parameter",
|
|
)
|
|
negative_style_prompt: Optional[str] = InputField(
|
|
default=None,
|
|
description="The negative style prompt parameter",
|
|
)
|
|
|
|
# SDXL Refiner
|
|
refiner_model: Optional[MainModelField] = InputField(
|
|
default=None,
|
|
description="The SDXL Refiner model used",
|
|
)
|
|
refiner_cfg_scale: Optional[float] = InputField(
|
|
default=None,
|
|
description="The classifier-free guidance scale parameter used for the refiner",
|
|
)
|
|
refiner_steps: Optional[int] = InputField(
|
|
default=None,
|
|
description="The number of steps used for the refiner",
|
|
)
|
|
refiner_scheduler: Optional[str] = InputField(
|
|
default=None,
|
|
description="The scheduler used for the refiner",
|
|
)
|
|
refiner_positive_aesthetic_score: Optional[float] = InputField(
|
|
default=None,
|
|
description="The aesthetic score used for the refiner",
|
|
)
|
|
refiner_negative_aesthetic_score: Optional[float] = InputField(
|
|
default=None,
|
|
description="The aesthetic score used for the refiner",
|
|
)
|
|
refiner_start: Optional[float] = InputField(
|
|
default=None,
|
|
description="The start value used for refiner denoising",
|
|
)
|
|
|
|
def invoke(self, context: InvocationContext) -> MetadataAccumulatorOutput:
|
|
"""Collects and outputs a CoreMetadata object"""
|
|
|
|
return MetadataAccumulatorOutput(metadata=CoreMetadata(**self.model_dump()))
|