- Update backend metadata for t2i adapter
- Fix typo in `T2IAdapterInvocation`: `ip_adapter_model` -> `t2i_adapter_model`
- Update linear graphs to use t2i adapter
- Add client metadata recall for t2i adapter
- Fix bug with controlnet metadata recall - processor should be set to 'none' when recalling a control adapter
Control adapters logic/state/ui is now generalized to hold controlnet, ip_adapter and t2i_adapter. In the future, other control adapter types can be added.
TODO:
- Limit IP adapter to 1
- Add T2I adapter to linear graphs
- Fix autoprocess
- T2I metadata saving & recall
- Improve on control adapters UI
This caused a crapload of network requests any time an image was generated.
The counts are necessary to handle the logic for inserting images into existing image list caches; we have to keep track of the counts.
Replace tag invalidation with manual cache updates in all cases, except the initial request (which is necessary to get the initial image counts).
One subtle change is to make the counts an object instead of a number. This is required for `immer` to handle draft states. This should be raised as a bug with RTK Query, as no error is thrown when attempting to update a primitive immer draft.
* feat(ui): max upscale pixels config
Add `maxUpscalePixels: number` to the app config. The number should be the *total* number of pixels eg `maxUpscalePixels: 4096 * 4096`.
If not provided, any size image may be upscaled.
If the config is provided, users will see be advised if their image is too large for either model, or told to switch to an x2 model if it's only too large for x4.
The message is via tooltip in the popover and via toast if the user uses the hotkey to upscale.
* feat(ui): "mayUpscale" -> "isAllowedToUpscale"
* add control net to useRecallParams
* got recall controlnets working
* fix metadata viewer controlnet
* fix type errors
* fix controlnet metadata viewer
* add ip adapter to metadata
* added ip adapter to recall parameters
* got ip adapter recall working, still need to fix type errors
* fix type issues
* clean up logs
* python formatting
* cleanup
* fix(ui): only store `image_name` as ip adapter image
* fix(ui): use nullish coalescing operator for numbers
Need to use the nullish coalescing operator `??` instead of false-y coalescing operator `||` when the value being check is a number. This prevents unintended coalescing when the value is zero and therefore false-y.
* feat(ui): fall back on default values for ip adapter metadata
* fix(ui): remove unused schema
* feat(ui): re-use existing schemas in metadata schema
* fix(ui): do not disable invocationCache
---------
Co-authored-by: psychedelicious <4822129+psychedelicious@users.noreply.github.com>
This hook was rerendering any time anything changed. Moved it to a logical component, put its useEffects inside the component. This reduces the effect of the rerenders to just that tiny always-null component.
* feat(ui): add error handling for enqueueBatch route, remove sessions
This re-implements the handling for the session create/invoke errors, but for batches.
Also remove all references to the old sessions routes in the UI.
* feat(ui): improve canvas image error UI
* make canvas error state gray instead of red
---------
Co-authored-by: Mary Hipp <maryhipp@Marys-MacBook-Air.local>
This is actually a platform-specific issue. `madge` is complaining about a circular dependency on a single file - `invokeai/frontend/web/src/features/queue/store/nanoStores.ts`. In that file, we import from the `nanostores` package. Very similar name to the file itself.
The error only appears on Windows and macOS, I imagine because those systems both resolve `nanostores` to itself before resolving to the package.
The solution is simple - rename `nanoStores.ts`. It's now `queueNanoStore.ts`.
- No longer need to make network request to add image to board after it's finished - removed
- Update linear graphs & upscale graph to save image to the board
- Update autoSwitch logic so when image is generated we still switch to the right board
* break out separate functions for preselected images, remove recallAllParameters dep as it causes circular logic with model being set
* lint
---------
Co-authored-by: Mary Hipp <maryhipp@Marys-MacBook-Air.local>
- New routes to clear, enable, disable and get the status of the cache
- Status includes hits, misses, size, max size, enabled
- Add client cache queries and mutations, abstracted into hooks
- Add invocation cache status area (next to queue status) w/ buttons
* feat(ui): tweak queue UI components
* fix(ui): manually dispatch queue status query on queue item status change
RTK Query occasionally aborts the query that occurs when the tag is invalidated, especially if multples of them fire in rapid succession.
This resulted in the queue status and progress bar sometimes not reseting when the queue finishes its last item.
Manually dispatch the query now to get around this. Eventually should probably move this to a socket so we don't need to keep responding to socket with HTTP requests. Just send ti directly via socket
* chore(ui): remove errant console.logs
* fix(ui): do not accumulate node outputs in outputs area
* fix(ui): fix merge issue
---------
Co-authored-by: Kent Keirsey <31807370+hipsterusername@users.noreply.github.com>
Add `batch_id` to outbound events. This necessitates adding it to both `InvocationContext` and `InvocationQueueItem`. This allows the canvas to receive images.
When the user enqueues a batch on the canvas, it is expected that all images from that batch are directed to the canvas.
The simplest, most flexible solution is to add the `batch_id` to the invocation context-y stuff. Then everything knows what batch it came from, and we can have the canvas pick up images associated with its list of canvas `batch_id`s.
* fix(config): fix typing issues in `config/`
`config/invokeai_config.py`:
- use `Optional` for things that are optional
- fix typing of `ram_cache_size()` and `vram_cache_size()`
- remove unused and incorrectly typed method `autoconvert_path`
- fix types and logic for `parse_args()`, in which `InvokeAIAppConfig.initconf` *must* be a `DictConfig`, but function would allow it to be set as a `ListConfig`, which presumably would cause issues elsewhere
`config/base.py`:
- use `cls` for first arg of class methods
- use `Optional` for things that are optional
- fix minor type issue related to setting of `env_prefix`
- remove unused `add_subparser()` method, which calls `add_parser()` on an `ArgumentParser` (method only available on the `_SubParsersAction` object, which is returned from ArgumentParser.add_subparsers()`)
* feat: queued generation and batches
Due to a very messy branch with broad addition of `isort` on `main` alongside it, some git surgery was needed to get an agreeable git history. This commit represents all of the work on queued generation. See PR for notes.
* chore: flake8, isort, black
* fix(nodes): fix incorrect service stop() method
* fix(nodes): improve names of a few variables
* fix(tests): fix up tests after changes to batches/queue
* feat(tests): add unit tests for session queue helper functions
* feat(ui): dynamic prompts is always enabled
* feat(queue): add queue_status_changed event
* feat(ui): wip queue graphs
* feat(nodes): move cleanup til after invoker startup
* feat(nodes): add cancel_by_batch_ids
* feat(ui): wip batch graphs & UI
* fix(nodes): remove `Batch.batch_id` from required
* fix(ui): cleanup and use fixedCacheKey for all mutations
* fix(ui): remove orphaned nodes from canvas graphs
* fix(nodes): fix cancel_by_batch_ids result count
* fix(ui): only show cancel batch tooltip when batches were canceled
* chore: isort
* fix(api): return `[""]` when dynamic prompts generates no prompts
Just a simple fallback so we always have a prompt.
* feat(ui): dynamicPrompts.combinatorial is always on
There seems to be little purpose in using the combinatorial generation for dynamic prompts. I've disabled it by hiding it from the UI and defaulting combinatorial to true. If we want to enable it again in the future it's straightforward to do so.
* feat: add queue_id & support logic
* feat(ui): fix upscale button
It prepends the upscale operation to queue
* feat(nodes): return queue item when enqueuing a single graph
This facilitates one-off graph async workflows in the client.
* feat(ui): move controlnet autoprocess to queue
* fix(ui): fix non-serializable DOMRect in redux state
* feat(ui): QueueTable performance tweaks
* feat(ui): update queue list
Queue items expand to show the full queue item. Just as JSON for now.
* wip threaded session_processor
* feat(nodes,ui): fully migrate queue to session_processor
* feat(nodes,ui): add processor events
* feat(ui): ui tweaks
* feat(nodes,ui): consolidate events, reduce network requests
* feat(ui): cleanup & abstract queue hooks
* feat(nodes): optimize batch permutation
Use a generator to do only as much work as is needed.
Previously, though we only ended up creating exactly as many queue items as was needed, there was still some intermediary work that calculated *all* permutations. When that number was very high, the system had a very hard time and used a lot of memory.
The logic has been refactored to use a generator. Additionally, the batch validators are optimized to return early and use less memory.
* feat(ui): add seed behaviour parameter
This dynamic prompts parameter allows the seed to be randomized per prompt or per iteration:
- Per iteration: Use the same seed for all prompts in a single dynamic prompt expansion
- Per prompt: Use a different seed for every single prompt
"Per iteration" is appropriate for exploring a the latents space with a stable starting noise, while "Per prompt" provides more variation.
* fix(ui): remove extraneous random seed nodes from linear graphs
* fix(ui): fix controlnet autoprocess not working when queue is running
* feat(queue): add timestamps to queue status updates
Also show execution time in queue list
* feat(queue): change all execution-related events to use the `queue_id` as the room, also include `queue_item_id` in InvocationQueueItem
This allows for much simpler handling of queue items.
* feat(api): deprecate sessions router
* chore(backend): tidy logging in `dependencies.py`
* fix(backend): respect `use_memory_db`
* feat(backend): add `config.log_sql` (enables sql trace logging)
* feat: add invocation cache
Supersedes #4574
The invocation cache provides simple node memoization functionality. Nodes that use the cache are memoized and not re-executed if their inputs haven't changed. Instead, the stored output is returned.
## Results
This feature provides anywhere some significant to massive performance improvement.
The improvement is most marked on large batches of generations where you only change a couple things (e.g. different seed or prompt for each iteration) and low-VRAM systems, where skipping an extraneous model load is a big deal.
## Overview
A new `invocation_cache` service is added to handle the caching. There's not much to it.
All nodes now inherit a boolean `use_cache` field from `BaseInvocation`. This is a node field and not a class attribute, because specific instances of nodes may want to opt in or out of caching.
The recently-added `invoke_internal()` method on `BaseInvocation` is used as an entrypoint for the cache logic.
To create a cache key, the invocation is first serialized using pydantic's provided `json()` method, skipping the unique `id` field. Then python's very fast builtin `hash()` is used to create an integer key. All implementations of `InvocationCacheBase` must provide a class method `create_key()` which accepts an invocation and outputs a string or integer key.
## In-Memory Implementation
An in-memory implementation is provided. In this implementation, the node outputs are stored in memory as python classes. The in-memory cache does not persist application restarts.
Max node cache size is added as `node_cache_size` under the `Generation` config category.
It defaults to 512 - this number is up for discussion, but given that these are relatively lightweight pydantic models, I think it's safe to up this even higher.
Note that the cache isn't storing the big stuff - tensors and images are store on disk, and outputs include only references to them.
## Node Definition
The default for all nodes is to use the cache. The `@invocation` decorator now accepts an optional `use_cache: bool` argument to override the default of `True`.
Non-deterministic nodes, however, should set this to `False`. Currently, all random-stuff nodes, including `dynamic_prompt`, are set to `False`.
The field name `use_cache` is now effectively a reserved field name and possibly a breaking change if any community nodes use this as a field name. In hindsight, all our reserved field names should have been prefixed with underscores or something.
## One Gotcha
Leaf nodes probably want to opt out of the cache, because if they are not cached, their outputs are not saved again.
If you run the same graph multiple times, you only end up with a single image output, because the image storage side-effects are in the `invoke()` method, which is bypassed if we have a cache hit.
## Linear UI
The linear graphs _almost_ just work, but due to the gotcha, we need to be careful about the final image-outputting node. To resolve this, a `SaveImageInvocation` node is added and used in the linear graphs.
This node is similar to `ImagePrimitive`, except it saves a copy of its input image, and has `use_cache` set to `False` by default.
This is now the leaf node in all linear graphs, and is the only node in those graphs with `use_cache == False` _and_ the only node with `is_intermedate == False`.
## Workflow Editor
All nodes now have a footer with a new `Use Cache [ ]` checkbox. It defaults to the value set by the invocation in its python definition, but can be changed by the user.
The workflow/node validation logic has been updated to migrate old workflows to use the new default values for `use_cache`. Users may still want to review the settings that have been chosen. In the event of catastrophic failure when running this migration, the default value of `True` is applied, as this is correct for most nodes.
Users should consider saving their workflows after loading them in and having them updated.
## Future Enhancements - Callback
A future enhancement would be to provide a callback to the `use_cache` flag that would be run as the node is executed to determine, based on its own internal state, if the cache should be used or not.
This would be useful for `DynamicPromptInvocation`, where the deterministic behaviour is determined by the `combinatorial: bool` field.
## Future Enhancements - Persisted Cache
Similar to how the latents storage is backed by disk, the invocation cache could be persisted to the database or disk. We'd need to be very careful about deserializing outputs, but it's perhaps worth exploring in the future.
* fix(ui): fix queue list item width
* feat(nodes): do not send the whole node on every generator progress
* feat(ui): strip out old logic related to sessions
Things like `isProcessing` are no longer relevant with queue. Removed them all & updated everything be appropriate for queue. May be a few little quirks I've missed...
* feat(ui): fix up param collapse labels
* feat(ui): click queue count to go to queue tab
* tidy(queue): update comment, query format
* feat(ui): fix progress bar when canceling
* fix(ui): fix circular dependency
* feat(nodes): bail on node caching logic if `node_cache_size == 0`
* feat(nodes): handle KeyError on node cache pop
* feat(nodes): bypass cache codepath if caches is disabled
more better no do thing
* fix(ui): reset api cache on connect/disconnect
* feat(ui): prevent enqueue when no prompts generated
* feat(ui): add queue controls to workflow editor
* feat(ui): update floating buttons & other incidental UI tweaks
* fix(ui): fix missing/incorrect translation keys
* fix(tests): add config service to mock invocation services
invoking needs access to `node_cache_size` to occur
* optionally remove pause/resume buttons from queue UI
* option to disable prepending
* chore(ui): remove unused file
* feat(queue): remove `order_id` entirely, `item_id` is now an autoinc pk
---------
Co-authored-by: Mary Hipp <maryhipp@Marys-MacBook-Air.local>
The immutable and serializable checks for redux can cause substantial performance issues. The immutable check in particular is pretty heavy. It's only run in dev mode, but this and really slow down the already-slower performance of dev mode.
The most important one for us is serializable, which has far less of a performance impact.
The immutable check is largely redundant because we use immer-backed RTK for everything and immer gives us confidence there.
Disable the immutable check, leaving serializable in.
JSX is not serializable, so it cannot be in redux. Non-serializable global state may be put into `nanostores`.
- Use `nanostores` for `customStarUI`
- Use `nanostores` for `headerComponent`
- Re-enable the serializable & immutable check redux middlewares
This simply hides nodes from the workflow editor. The nodes will still work if an API request is made with them. For example, you could hide `iterate` nodes from the workflow editor, but if the Linear UI makes use of those nodes, they will still function.
- Update `AppConfig` with optional property `nodesDenylist: string[]`
- If provided, nodes are filtered out by `type` in the workflow editor
- Node versions are now added to node templates
- Node data (including in workflows) include the version of the node
- On loading a workflow, we check to see if the node and template versions match exactly. If not, a warning is logged to console.
- The node info icon (top-right corner of node, which you may click to open the notes editor) now shows the version and mentions any issues.
- Some workflow validation logic has been shifted around and is now executed in a redux listener.
Adds loading workflows with exhaustive validation via `zod`.
There is a load button but no dedicated save/load UI yet. Also need to add versioning to the workflow format itself.
Previously if an image was used in nodes and you deleted it, it would reset all of node editor. Same for controlnet.
Now it only resets the specific nodes or controlnets that used that image.
- also implement pessimistic updates for starring, only changing the images that were successfully updated by backend
- some autoformat changes crept in
There was no check at all to see if the canvas had a valid model already selected. The first model in the list was selected every time.
Now, we check if its valid. If not, we go through the logic to try and pick the first valid model.
If there are no valid models, or there was a problem listing models, the model selection is cleared.
multi-select actions include:
- drag to board to move all to that board
- right click to add all to board or delete all
backend changes:
- add routes for changing board for list of image names, deleting list of images
- change image-specific routes to `images/i/{image_name}` to not clobber other routes (like `images/upload`, `images/delete`)
- subclass pydantic `BaseModel` as `BaseModelExcludeNull`, which excludes null values when calling `dict()` on the model. this fixes inconsistent types related to JSON parsing null values into `null` instead of `undefined`
- remove `board_id` from `remove_image_from_board`
frontend changes:
- multi-selection stuff uses `ImageDTO[]` as payloads, for dnd and other mutations. this gives us access to image `board_id`s when hitting routes, and enables efficient cache updates.
- consolidate change board and delete image modals to handle single and multiples
- board totals are now re-fetched on mutation and not kept in sync manually - was way too tedious to do this
- fixed warning about nested `<p>` elements
- closes#4088 , need to handle case when `autoAddBoardId` is `"none"`
- add option to show gallery image delete button on every gallery image
frontend refactors/organisation:
- make typegen script js instead of ts
- enable `noUncheckedIndexedAccess` to help avoid bugs when indexing into arrays, many small changes needed to satisfy TS after this
- move all image-related endpoints into `endpoints/images.ts`, its a big file now, but this fixes a number of circular dependency issues that were otherwise felt impossible to resolve
* add upper bound for minWidth to prevent crash with cypress
* add fallback so UI doesnt crash when backend isnt running
---------
Co-authored-by: Mary Hipp <maryhipp@Marys-MacBook-Air.local>
- add `addNSFWCheckerToGraph` and `addWatermarkerToGraph` functions
- use them in all linear graph creation
- add state & toggles to settings modal to enable these
- trigger queries for app config on socket connect
- disable the nsfw/watermark booleans if we get the app config and they are not available
Rolled back the earlier split of the refiner model query.
Now, when you use `useGetMainModelsQuery()`, you must provide it an array of base model types.
They are provided as constants for simplicity:
- ALL_BASE_MODELS
- NON_REFINER_BASE_MODELS
- REFINER_BASE_MODELS
Opted to just use args for the hook instead of wrapping the hook in another hook, we can tidy this up later if desired.
We can derive `isRefinerAvailable` from the query result (eg are there any refiner models installed). This is a piece of server state, so by using the list models response directly, we can avoid needing to manually keep the client in sync with the server.
Created a `useIsRefinerAvailable()` hook to return this boolean wherever it is needed.
Also updated the main models & refiner models endpoints to only return the appropriate models. Now we don't need to filter the data on these endpoints.
At some point I typo'd this and set the max seed to signed int32 max. It should be *un*signed int32 max.
This restored the seed range to what it was in v2.3.
- use the existing logic to determine if generation is txt2img, img2img, inpaint or outpaint
- technically `outpaint` and `inpaint` are the same, just display
"Inpaint" if its either
- debounce this by 1s to prevent jank
When a queue item is popped for processing, we need to retrieve its session from the DB. Pydantic serializes the graph at this stage.
It's possible for a graph to have been made invalid during the graph preparation stage (e.g. an ancestor node executes, and its output is not valid for its successor node's input field).
When this occurs, the session in the DB will fail validation, but we don't have a chance to find out until it is retrieved and parsed by pydantic.
This logic was previously not wrapped in any exception handling.
Just after retrieving a session, we retrieve the specific invocation to execute from the session. It's possible that this could also have some sort of error, though it should be impossible for it to be a pydantic validation error (that would have been caught during session validation). There was also no exception handling here.
When either of these processes fail, the processor gets soft-locked because the processor's cleanup logic is never run. (I didn't dig deeper into exactly what cleanup is not happening, because the fix is to just handle the exceptions.)
This PR adds exception handling to both the session retrieval and node retrieval and events for each: `session_retrieval_error` and `invocation_retrieval_error`.
These events are caught and displayed in the UI as toasts, along with the type of the python exception (e.g. `Validation Error`). The events are also logged to the browser console.
* feat(ui): enhance clear intermediates feature
- retrieve the # of intermediates using a new query (just uses list images endpoint w/ limit of 0)
- display the count in the UI
- add types for clearIntermediates mutation
- minor styling and verbiage changes
* feat(ui): remove unused settings option for guides
* feat(ui): use solid badge variant
consistent with the rest of the usage of badges
* feat(ui): update board ctx menu, add board auto-add
- add context menu to system boards - only open is select board. did this so that you dont think its broken when you click it
- add auto-add board. you can right click a user board to enable it for auto-add, or use the gallery settings popover to select it. the invoke button has a tooltip on a short delay to remind you that you have auto-add enabled
- made useBoardName hook, provide it a board id and it gets your the board name
- removed `boardIdToAdTo` state & logic, updated workflows to auto-switch and auto-add on image generation
* fix(ui): clear controlnet when clearing intermediates
* feat: Make Add Board icon a button
* feat(db, api): clear intermediates now clears all of them
* feat(ui): make reset webui text subtext style
* feat(ui): board name change submits on blur
---------
Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com>
* feat(ui): migrate listImages to RTK query using createEntityAdapter
- see comments in `endpoints/images.ts` for explanation of the caching
- so far, only manually updating `all` images when new image is generated. no other manual cache updates are implemented, but will be needed.
- fixed some weirdness with loading state components (like the spinners in gallery)
- added `useThumbnailFallback` for `IAIDndImage`, this displays the tiny webp thumbnail while the full-size images load
- comment out some old thunk related stuff in gallerySlice, which is no longer needed
* feat(ui): add manual cache updates for board changes (wip)
- update RTK Query caches when adding/removing single image to/from board
- work more on migrating all image-related operations to RTK Query
* update AddImagesToBoardContext so that it works when user uses context menu + modal
* handle case where no image is selected
* get assets working for main list and boards - dnd only
* feat(ui): migrate image uploads to RTK Query
- minor refactor of `ImageUploader` and `useImageUploadButton` hooks, simplify some logic
- style filesystem upload overlay to match existing UI
- replace all old `imageUploaded` thunks with `uploadImage` RTK Query calls, update associated logic including canvas related uploads
- simplify `PostUploadAction`s that only need to display user input
* feat(ui): remove `receivedPageOfImages` thunks
* feat(ui): remove `receivedImageUrls` thunk
* feat(ui): finish removing all images thunks
stuff now broken:
- image usage
- delete board images
- on first load, no image selected
* feat(ui): simplify `updateImage` cache manipulation
- we don't actually ever change categories, so we can remove a lot of logic
* feat(ui): simplify canvas autosave
- instead of using a network request to set the canvas generation as not intermediate, we can just do that in the graph
* feat(ui): simplify & handle edge cases in cache updates
* feat(db, api): support `board_id='none'` for `get_many` images queries
This allows us to get all images that are not on a board.
* chore(ui): regen types
* feat(ui): add `All Assets`, `No Board` boards
Restructure boards:
- `all images` is all images
- `all assets` is all assets
- `no board` is all images/assets without a board set
- user boards may have images and assets
Update caching logic
- much simpler without every board having sub-views of images and assets
- update drag and drop operations for all possible interactions
* chore(ui): regen types
* feat(ui): move download to top of context menu
* feat(ui): improve drop overlay styles
* fix(ui): fix image not selected on first load
- listen for first load of all images board, then select the first image
* feat(ui): refactor board deletion
api changes:
- add route to list all image names for a board. this is required to handle board + image deletion. we need to know every image in the board to determine the image usage across the app. this is fetched only when the delete board and images modal is opened so it's as efficient as it can be.
- update the delete board route to respond with a list of deleted `board_images` and `images`, as image names. this is needed to perform accurate clientside state & cache updates after deleting.
db changes:
- remove unused `board_images` service method to get paginated images dtos for a board. this is now done thru the list images endpoint & images service. needs a small logic change on `images.delete_images_on_board`
ui changes:
- simplify the delete board modal - no context, just minor prop drilling. this is feasible for boards only because the components that need to trigger and manipulate the modal are very close together in the tree
- add cache updates for `deleteBoard` & `deleteBoardAndImages` mutations
- the only thing we cannot do directly is on `deleteBoardAndImages`, update the `No Board` board. we'd need to insert image dtos that we may not have loaded. instead, i am just invalidating the tags for that `listImages` cache. so when you `deleteBoardAndImages`, the `No Board` will re-fetch the initial image limit. i think this is more efficient than e.g. fetching all image dtos to insert then inserting them.
- handle image usage for `deleteBoardAndImages`
- update all (i think/hope) the little bits and pieces in the UI to accomodate these changes
* fix(ui): fix board selection logic
* feat(ui): add delete board modal loading state
* fix(ui): use thumbnails for board cover images
* fix(ui): fix race condition with board selection
when selecting a board that doesn't have any images loaded, we need to wait until the images haveloaded before selecting the first image.
this logic is debounced to ~1000ms.
* feat(ui): name 'No Board' correctly, change icon
* fix(ui): do not cache listAllImageNames query
if we cache it, we can end up with stale image usage during deletion.
we could of course manually update the cache as we are doing elsewhere. but because this is a relatively infrequent network request, i'd like to trade increased cache mgmt complexity here for increased resource usage.
* feat(ui): reduce drag preview opacity, remove border
* fix(ui): fix incorrect queryArg used in `deleteImage` and `updateImage` cache updates
* fix(ui): fix doubled open in new tab
* fix(ui): fix new generations not getting added to 'No Board'
* fix(ui): fix board id not changing on new image when autosave enabled
* fix(ui): context menu when selection is 0
need to revise how context menu is triggered later, when we approach multi select
* fix(ui): fix deleting does not update counts for all images and all assets
* fix(ui): fix all assets board name in boards list collapse button
* fix(ui): ensure we never go under 0 for total board count
* fix(ui): fix text overflow on board names
---------
Co-authored-by: Mary Hipp <maryhipp@Marys-MacBook-Air.local>
- update controlnet state to use object format for model
- update model-parsing helper functions to log errors
- update nodes components, types and state
- remove controlnets from state when models are loaded and the controlnet's model is not available
IAIMantineSelect and IAIMantineMultiSelect have a bit of extra logic that prevents simple select functionality from working as expected.
- extract the styles into hooks
- rename those two components to IAIMantineSearchableSelect and IAIMantineSearchableMultiSelect
- Create IAIMantineSelect (which is just a dropdown) and use it in model manager and a few other places
When we only have a few options to present and searching is not efficient, we should use this instead.
We were storing all types of models by their model ID, which is a format like `sd-1/main/deliberate`.
This meant we had to do a lot of extra parsing, because nodes actually wants something like `{base_model: 'sd-1', model_name: 'deliberate'}`.
Some of this parsing was done with zod's error-throwing `parse()` method, and in other places it was done with brittle string parsing.
This commit refactors the state to use the object form of models.
There is still a bit of string parsing done in the to construct the ID from the object form, but it's far less complicated.
Also, the zod parsing is now done using `safeParse()`, which does not throw. This requires a few more conditional checks, but should prevent further crashes.
* feat(ui): salvaged gallery UI enhancements
* restore boardimage functionality, load boardimages and remove some cachine optimizations in the name of data integrity
* fix assets, fix load more params
* jk NOW fix assets, fix load more params
---------
Co-authored-by: Mary Hipp <maryhipp@Marys-MacBook-Air.local>
Co-authored-by: Mary Hipp Rogers <maryhipp@gmail.com>
- available infill methods is server state - remove it from client state, use the query to populate the dropdown
- add listener to ensure the selected infill method is an available one
- Restore recall functionality to `CurrentImageButtons` and `ImageContextMenu`.
- Debounce metadata requests for `ImageMetadataViewer` and `CurrentImageButtons` by 500ms. It's possible to scroll through these really fast, so we want to debounce the network requests.
- `ImageContextMenu` is lazy-mounted so it does not need to be debounced; it makes the metadata request as soon as you click it.
- Move next/prev image selection logic into hook and add the hotkeys for this to `CurrentImageButtons`. The hotkeys now work when metadata viewer is open.
I will follow up with improved loading state during the debounced calls in the future
- Update for new routes
- Update model storage in state to be `MainModelField` type instead of `string`, simplifies a lot of model handling
- Update model-related stuff for model `name` --> `model_name`
- Update linear graphs to use `MetadataAccumulator`
- Update `ImageMetadataViewer` UI
- Ensure all `recall` functions work (well, the ones that are active anyways)
- `isLoading` - now `true` *only* on first load
- added `isFetching` - `true` whenever gallery images are fetching
- on first load, show a spinner instead of skeletons. this prevents an awkward flash of skeletons into empty gallery when the gallery doesn't have enough images to fill it.
- removed `imageCategoriesChanged` listener, bc now on app start, both images and assets will be populated. leaving this in caused jank flashes of skeletons when switching gallery tabs when gallery doesn't have images to load
taking the coward's way out on this and just fetching 100 images & 100 assets on app start...
- add `appStarted` action, dispatched once on mount in App.tsx. listener fetches 100 images & 100 assets
- fix bug with selectedBoardId & assets tab
The shift key listener didn't catch pressed when focused in a textarea or input field, causing jank on slider number inputs.
Add keydown and keyup listeners to all such fields, which ensures that the `shift` state is always correct.
Also add the action tracking it to `actionsDenylist` to not clutter up devtools.
* load images on gallery render
* wait for models to be loaded before you can invoke
---------
Co-authored-by: Mary Hipp <maryhipp@Marys-MacBook-Air.local>
This caused a lot of re-rendering whenever the selection changed, which caused a huge performance hit. It also made changing the current image lag a bit.
Instead of providing an array of image names as a multi-select dnd payload, there is now no multi-select dnd payload at all - instead, the payload types are used by the `imageDropped` listener to pull the selection out of redux.
Now, the only big re-renders are when the selectionCount changes. In the future I'll figure out a good way to do image names as payload without incurring re-renders.
This introduces the core functionality for batch operations on images and multiple selection in the gallery/batch manager.
A number of other substantial changes are included:
- `imagesSlice` is consolidated into `gallerySlice`, allowing for simpler selection of filtered images
- `batchSlice` is added to manage the batch
- The wonky context pattern for image deletion has been changed, much simpler now using a `imageDeletionSlice` and redux listeners; this needs to be implemented still for the other image modals
- Minimum gallery size in px implemented as a hook
- Many style fixes & several bug fixes
TODO:
- The UI and UX need to be figured out, especially for controlnet
- Batch processing is not hooked up; generation does not do anything with batch
- Routes to support batch image operations, specifically delete and add/remove to/from boards
Themes are very fun but due to the differences in perceived saturation and lightness across the
the color spectrum, it's impossible to have have multiple themes that look great without hand-
crafting *every* shade for *every* theme. We've ended up with 4 OK themes (well, 3, because the
light theme was pretty bad).
I've removed the themes and added color mode support. There is now a single dark and light mode,
each with their own color palette and the classic grey / purple / yellow invoke colors that
@blessedcoolant first designed.
I've re-styled almost everything except the model manager and lightbox, which I keep forgetting
to work on.
One new concept is the Chakra `layerStyle`. This lets us define "layers" - think body, first layer,
second layer, etc - that can be applied on various components. By defining layers, we can be more
consistent about the z-axis and its relationship to color and lightness.
Everything seems to be working.
- Due to a change to `reactflow`, I regenerated `yarn.lock`
- New chakra CLI fixes issue I had made a patch for; removed the patch
- Change to fontsource changed how we import that font
- Change to fontawesome means we lost the txt2img tab icon, just chose a similar one
*migrate from `openapi-typescript-codegen` to `openapi-typescript` and `openapi-fetch`*
`openapi-typescript-codegen` is not very actively maintained - it's been over a year since the last update.
`openapi-typescript` and `openapi-fetch` are part of the actively maintained repo. key differences:
- provides a `fetch` client instead of `axios`, which means we need to be a bit more verbose with typing thunks
- fetch client is created at runtime and has a very nice typescript DX
- generates a single file with all types in it, from which we then extract individual types. i don't like how verbose this is, but i do like how it is more explicit.
- removed npm api generation scripts - now we have a single `typegen` script
overall i have more confidence in this new library.
*use nanostores for api base and token*
very simple reactive store for api base url and token. this was suggested in the `openapi-fetch` docs and i quite like the strategy.
*organise rtk-query api*
split out each endpoint (models, images, boards, boardImages) into their own api extensions. tidy!
Basically updated all slices to be more descriptive in their names. Did so in order to make sure theres good naming scheme available for secondary models.
To determine whether the Load More button should work, we need to keep track of how many images are left to load for a given board or category.
The Assets tab doesn't work, though. Need to figure out a better way to handle this.
We need to access the initial image dimensions during the creation of the `ImageToImage` graph to determine if we need to resize the image.
Because the `initialImage` is now just an image name, we need to either store (easy) or dynamically retrieve its dimensions during graph creation (a bit less easy).
Took the easiest path. May need to revise this in the future.
Images that are used as parameters (e.g. init image, canvas images) are stored as full `ImageDTO` objects in state, separate from and duplicating any object representing those same objects in the `imagesSlice`.
We cannot store only image names as parameters, then pull the full `ImageDTO` from `imagesSlice`, because if an image is not on a loaded page, it doesn't exist in `imagesSlice`. For example, if you scroll down a few pages in the gallery and send that image to canvas, on reloading the app, the canvas will be unable to load that image.
We solved this temporarily by storing the full `ImageDTO` object wherever it was needed, but this is both inefficient and allows for stale `ImageDTO`s across the app.
One other possible solution was to just fetch the `ImageDTO` for all images at startup, and insert them into the `imagesSlice`, but then we run into an issue where we are displaying images in the gallery totally out of context.
For example, if an image from several pages into the gallery was sent to canvas, and the user refreshes, we'd display the first 20 images in gallery. Then to populate the canvas, we'd fetch that image we sent to canvas and add it to `imagesSlice`. Now we'd have 21 images in the gallery: 1 to 20 and whichever image we sent to canvas. Weird.
Using `rtk-query` solves this by allowing us to very easily fetch individual images in the components that need them, and not directly interact with `imagesSlice`.
This commit changes all references to images-as-parameters to store only the name of the image, and not the full `ImageDTO` object. Then, we use an `rtk-query` generated `useGetImageDTOQuery()` hook in each of those components to fetch the image.
We can use cache invalidation when we mutate any image to trigger automated re-running of the query and all the images are automatically kept up to date.
This also obviates the need for the convoluted URL fetching scheme for images that are used as parameters. The `imagesSlice` still need this handling unfortunately.
- Add graph builders for canvas txt2img & img2img - they are mostly copy and paste from the linear graph builders but different in a few ways that are very tricky to work around. Just made totally new functions for them.
- Canvas txt2img and img2img support ControlNet (not inpaint/outpaint). There's no way to determine in real-time which mode the canvas is in just yet, so we cannot disable the ControlNet UI when the mode will be inpaint/outpaint - it will always display. It's possible to determine this in near-real-time, will add this at some point.
- Canvas inpaint/outpaint migrated to use model loader, though inpaint/outpaint are still using the non-latents nodes.
Instead of manually creating every node and edge, we can simply copy/paste the base graph from node editor, then sub in parameters.
This is a much more intelligible process. We still need to handle seed, img2img fit and controlnet separately.
- remove UI-specific state (the enabled schedulers) from redux, instead derive it in a selector
- simplify logic by putting schedulers in an object instead of an array
- rename `activeSchedulers` to `enabledSchedulers`
- remove need for `useEffect()` when `enabledSchedulers` changes by adding a listener for the `enabledSchedulersChanged` action/event to `generationSlice`
- increase type safety by making `enabledSchedulers` an array of `SchedulerParam`, which is created by the zod schema for scheduler
The processor is automatically selected when model is changed.
But if the user manually changes the processor, processor settings, or disables the new `Auto configure processor` switch, auto processing is disabled.
The user can enable auto configure by turning the switch back on.
When auto configure is enabled, a small dot is overlaid on the expand button to remind the user that the system is not auto configuring the processor for them.
If auto configure is enabled, the processor settings are reset to the default for the selected model.
Add uploading to IAIDndImage
- add `postUploadAction` arg to `imageUploaded` thunk, with several current valid options (set control image, set init, set nodes image, set canvas, or toast)
- updated IAIDndImage to optionally allow click to upload
This handles the case when an image is deleted but is still in use in as eg an init image on canvas, or a control image. If we just delete the image, canvas/controlnet/etc may break (the image would just fail to load).
When an image is deleted, the app checks to see if it is in use in:
- Image to Image
- ControlNet
- Unified Canvas
- Node Editor
The delete dialog will always open if the image is in use anywhere, and the user is advised that deleting the image will reset the feature(s).
Even if the user has ticked the box to not confirm on delete, the dialog will still show if the image is in use somewhere.
- fix "bounding box region only" not being respected when saving
- add toasts for each action
- improve workflow `take()` predicates to use the requestId
- responsive changes were causing a lot of weird layout issues, had to remove the rest of them
- canvas (non-beta) toolbar now wraps
- reduces minH for prompt boxes a bit
Implement `dnd-kit` for image drag and drop
- vastly simplifies logic bc we can drag and drop non-serializable data (like an `ImageDTO`)
- also much prettier
- also will fix conflicts with file upload via OS drag and drop, bc `dnd-kit` does not use native HTML drag and drop API
- Implemented for Init image, controlnet, and node editor so far
More progress on the ControlNet UI
Some socket events should not be handled by the slice reducers. For example generation progress should not be handled for a canceled session.
Added another layer of socket actions.
Example:
- `socketGeneratorProgress` is dispatched when the actual socket event is received
- Listener middleware exclusively handles this event and determines if the application should also handle it
- If so, it dispatches `appSocketGeneratorProgress`, which the slices can handle
Needed to fix issues related to canceling invocations.
Now that images are in a database and we can make filtered queries, we can do away with the cumbersome `resultsSlice` and `uploadsSlice`.
- Remove `resultsSlice` and `uploadsSlice` entirely
- Add `imagesSlice` fills the same role
- Convert the application to use `imagesSlice`, reducing a lot of messy logic where we had to check which category was selected
- Add a simple filter popover to the gallery, which lets you select any number of image categories
- Update all thunks & network related things
- Update gallery
What I have not done yet is rename the gallery tabs and the relevant slices, but I believe the functionality is all there.
Also I fixed several bugs along the way but couldn't really commit them separately bc I was refactoring. Can't remember what they were, but related to the gallery image switching.
- Update the canvas graph generation to flag its uploaded init and mask images as `intermediate`.
- During canvas setup, hit the update route to associate the uploaded images with the session id.
- Organize the socketio and RTK listener middlware better. Needed to facilitate the updated canvas logic.
- Add a new action `sessionReadyToInvoke`. The `sessionInvoked` action is *only* ever run in response to this event. This lets us do whatever complicated setup (eg canvas) and explicitly invoking. Previously, invoking was tied to the socket subscribe events.
- Some minor tidying.
* added optional middleware prop and new actions needed
* accidental import
* make middleware an array
---------
Co-authored-by: Mary Hipp <maryhipp@Marys-MacBook-Air.local>
