mirror of
https://github.com/invoke-ai/InvokeAI
synced 2024-08-30 20:32:17 +00:00
Partial migration of UI to nodes API (#3195)
* feat(ui): add axios client generator and simple example * fix(ui): update client & nodes test code w/ new Edge type * chore(ui): organize generated files * chore(ui): update .eslintignore, .prettierignore * chore(ui): update openapi.json * feat(backend): fixes for nodes/generator * feat(ui): generate object args for api client * feat(ui): more nodes api prototyping * feat(ui): nodes cancel * chore(ui): regenerate api client * fix(ui): disable OG web server socket connection * fix(ui): fix scrollbar styles typing and prop just noticed the typo, and made the types stronger. * feat(ui): add socketio types * feat(ui): wip nodes - extract api client method arg types instead of manually declaring them - update example to display images - general tidy up * start building out node translations from frontend state and add notes about missing features * use reference to sampler_name * use reference to sampler_name * add optional apiUrl prop * feat(ui): start hooking up dynamic txt2img node generation, create middleware for session invocation * feat(ui): write separate nodes socket layer, txt2img generating and rendering w single node * feat(ui): img2img implementation * feat(ui): get intermediate images working but types are stubbed out * chore(ui): add support for package mode * feat(ui): add nodes mode script * feat(ui): handle random seeds * fix(ui): fix middleware types * feat(ui): add rtk action type guard * feat(ui): disable NodeAPITest This was polluting the network/socket logs. * feat(ui): fix parameters panel border color This commit should be elsewhere but I don't want to break my flow * feat(ui): make thunk types more consistent * feat(ui): add type guards for outputs * feat(ui): load images on socket connect Rudimentary * chore(ui): bump redux-toolkit * docs(ui): update readme * chore(ui): regenerate api client * chore(ui): add typescript as dev dependency I am having trouble with TS versions after vscode updated and now uses TS 5. `madge` has installed 3.9.10 and for whatever reason my vscode wants to use that. Manually specifying 4.9.5 and then setting vscode to use that as the workspace TS fixes the issue. * feat(ui): begin migrating gallery to nodes Along the way, migrate to use RTK `createEntityAdapter` for gallery images, and separate `results` and `uploads` into separate slices. Much cleaner this way. * feat(ui): clean up & comment results slice * fix(ui): separate thunk for initial gallery load so it properly gets index 0 * feat(ui): POST upload working * fix(ui): restore removed type * feat(ui): patch api generation for headers access * chore(ui): regenerate api * feat(ui): wip gallery migration * feat(ui): wip gallery migration * chore(ui): regenerate api * feat(ui): wip refactor socket events * feat(ui): disable panels based on app props * feat(ui): invert logic to be disabled * disable panels when app mounts * feat(ui): add support to disableTabs * docs(ui): organise and update docs * lang(ui): add toast strings * feat(ui): wip events, comments, and general refactoring * feat(ui): add optional token for auth * feat(ui): export StatusIndicator and ModelSelect for header use * feat(ui) working on making socket URL dynamic * feat(ui): dynamic middleware loading * feat(ui): prep for socket jwt * feat(ui): migrate cancelation also updated action names to be event-like instead of declaration-like sorry, i was scattered and this commit has a lot of unrelated stuff in it. * fix(ui): fix img2img type * chore(ui): regenerate api client * feat(ui): improve InvocationCompleteEvent types * feat(ui): increase StatusIndicator font size * fix(ui): fix middleware order for multi-node graphs * feat(ui): add exampleGraphs object w/ iterations example * feat(ui): generate iterations graph * feat(ui): update ModelSelect for nodes API * feat(ui): add hi-res functionality for txt2img generations * feat(ui): "subscribe" to particular nodes feels like a dirty hack but oh well it works * feat(ui): first steps to node editor ui * fix(ui): disable event subscription it is not fully baked just yet * feat(ui): wip node editor * feat(ui): remove extraneous field types * feat(ui): nodes before deleting stuff * feat(ui): cleanup nodes ui stuff * feat(ui): hook up nodes to redux * fix(ui): fix handle * fix(ui): add basic node edges & connection validation * feat(ui): add connection validation styling * feat(ui): increase edge width * feat(ui): it blends * feat(ui): wip model handling and graph topology validation * feat(ui): validation connections w/ graphlib * docs(ui): update nodes doc * feat(ui): wip node editor * chore(ui): rebuild api, update types * add redux-dynamic-middlewares as a dependency * feat(ui): add url host transformation * feat(ui): handle already-connected fields * feat(ui): rewrite SqliteItemStore in sqlalchemy * fix(ui): fix sqlalchemy dynamic model instantiation * feat(ui, nodes): metadata wip * feat(ui, nodes): models * feat(ui, nodes): more metadata wip * feat(ui): wip range/iterate * fix(nodes): fix sqlite typing * feat(ui): export new type for invoke component * tests(nodes): fix test instantiation of ImageField * feat(nodes): fix LoadImageInvocation * feat(nodes): add `title` ui hint * feat(nodes): make ImageField attrs optional * feat(ui): wip nodes etc * feat(nodes): roll back sqlalchemy * fix(nodes): partially address feedback * fix(backend): roll back changes to pngwriter * feat(nodes): wip address metadata feedback * feat(nodes): add seeded rng to RandomRange * feat(nodes): address feedback * feat(nodes): move GET images error handling to DiskImageStorage * feat(nodes): move GET images error handling to DiskImageStorage * fix(nodes): fix image output schema customization * feat(ui): img2img/txt2img -> linear - remove txt2img and img2img tabs - add linear tab - add initial image selection to linear parameters accordion * feat(ui): tidy graph builders * feat(ui): tidy misc * feat(ui): improve invocation union types * feat(ui): wip metadata viewer recall * feat(ui): move fonts to normal deps * feat(nodes): fix broken upload * feat(nodes): add metadata module + tests, thumbnails - `MetadataModule` is stateless and needed in places where the `InvocationContext` is not available, so have not made it a `service` - Handles loading/parsing/building metadata, and creating png info objects - added tests for MetadataModule - Lifted thumbnail stuff to util * fix(nodes): revert change to RandomRangeInvocation * feat(nodes): address feedback - make metadata a service - rip out pydantic validation, implement metadata parsing as simple functions - update tests - address other minor feedback items * fix(nodes): fix other tests * fix(nodes): add metadata service to cli * fix(nodes): fix latents/image field parsing * feat(nodes): customise LatentsField schema * feat(nodes): move metadata parsing to frontend * fix(nodes): fix metadata test --------- Co-authored-by: maryhipp <maryhipp@gmail.com> Co-authored-by: Mary Hipp <maryhipp@Marys-MacBook-Air.local>
This commit is contained in:
psychedelicious
GitHub
87
invokeai/frontend/web/docs/API_CLIENT.md
Normal file
87
invokeai/frontend/web/docs/API_CLIENT.md
Normal file
@ -0,0 +1,87 @@
|
||||
# Generated axios API client
|
||||
|
||||
- [Generated axios API client](#generated-axios-api-client)
|
||||
- [Generation](#generation)
|
||||
- [Generate the API client from the nodes web server](#generate-the-api-client-from-the-nodes-web-server)
|
||||
- [Generate the API client from JSON](#generate-the-api-client-from-json)
|
||||
- [Getting the JSON from the nodes web server](#getting-the-json-from-the-nodes-web-server)
|
||||
- [Getting the JSON with a python script](#getting-the-json-with-a-python-script)
|
||||
- [Generate the API client](#generate-the-api-client)
|
||||
- [The generated client](#the-generated-client)
|
||||
- [API client customisation](#api-client-customisation)
|
||||
|
||||
This API client is generated by an [openapi code generator](https://github.com/ferdikoomen/openapi-typescript-codegen).
|
||||
|
||||
All files in `invokeai/frontend/web/src/services/api/` are made by the generator.
|
||||
|
||||
## Generation
|
||||
|
||||
The axios client may be generated by from the OpenAPI schema from the nodes web server, or from JSON.
|
||||
|
||||
### Generate the API client from the nodes web server
|
||||
|
||||
We need to start the nodes web server, which serves the OpenAPI schema to the generator.
|
||||
|
||||
1. Start the nodes web server.
|
||||
|
||||
```bash
|
||||
# from the repo root
|
||||
python scripts/invoke-new.py --web
|
||||
```
|
||||
|
||||
2. Generate the API client.
|
||||
|
||||
```bash
|
||||
# from invokeai/frontend/web/
|
||||
yarn api:web
|
||||
```
|
||||
|
||||
### Generate the API client from JSON
|
||||
|
||||
The JSON can be acquired from the nodes web server, or with a python script.
|
||||
|
||||
#### Getting the JSON from the nodes web server
|
||||
|
||||
Start the nodes web server as described above, then download the file.
|
||||
|
||||
```bash
|
||||
# from invokeai/frontend/web/
|
||||
curl http://localhost:9090/openapi.json -o openapi.json
|
||||
```
|
||||
|
||||
#### Getting the JSON with a python script
|
||||
|
||||
Run this python script from the repo root, so it can access the nodes server modules.
|
||||
|
||||
The script will output `openapi.json` in the repo root. Then we need to move it to `invokeai/frontend/web/`.
|
||||
|
||||
```bash
|
||||
# from the repo root
|
||||
python invokeai/app/util/generate_openapi_json.py
|
||||
mv invokeai/app/util/openapi.json invokeai/frontend/web/services/fixtures/
|
||||
```
|
||||
|
||||
#### Generate the API client
|
||||
|
||||
Now we can generate the API client from the JSON.
|
||||
|
||||
```bash
|
||||
# from invokeai/frontend/web/
|
||||
yarn api:file
|
||||
```
|
||||
|
||||
## The generated client
|
||||
|
||||
The client will be written to `invokeai/frontend/web/services/api/`:
|
||||
|
||||
- `axios` client
|
||||
- TS types
|
||||
- An easily parseable schema, which we can use to generate UI
|
||||
|
||||
## API client customisation
|
||||
|
||||
The generator has a default `request.ts` file that implements a base `axios` client. The generated client uses this base client.
|
||||
|
||||
One shortcoming of this is base client is it does not provide response headers unless the response body is empty. To fix this, we provide our own lightly-patched `request.ts`.
|
||||
|
||||
To access the headers, call `getHeaders(response)` on any response from the generated api client. This function is exported from `invokeai/frontend/web/src/services/util/getHeaders.ts`.
|
||||
21
invokeai/frontend/web/docs/EVENTS.md
Normal file
21
invokeai/frontend/web/docs/EVENTS.md
Normal file
@ -0,0 +1,21 @@
|
||||
# Events
|
||||
|
||||
Events via `socket.io`
|
||||
|
||||
## `actions.ts`
|
||||
|
||||
Redux actions for all socket events. Payloads all include a timestamp, and optionally some other data.
|
||||
|
||||
Any reducer (or middleware) can respond to the actions.
|
||||
|
||||
## `middleware.ts`
|
||||
|
||||
Redux middleware for events.
|
||||
|
||||
Handles dispatching the event actions. Only put logic here if it can't really go anywhere else.
|
||||
|
||||
For example, on connect we want to load images to the gallery if it's not populated. This requires dispatching a thunk, so we need to directly dispatch this in the middleware.
|
||||
|
||||
## `types.ts`
|
||||
|
||||
Hand-written types for the socket events. Cannot generate these from the server, but fortunately they are few and simple.
|
||||
17
invokeai/frontend/web/docs/NODE_EDITOR.md
Normal file
17
invokeai/frontend/web/docs/NODE_EDITOR.md
Normal file
@ -0,0 +1,17 @@
|
||||
# Node Editor Design
|
||||
|
||||
WIP
|
||||
|
||||
nodes
|
||||
|
||||
everything in `src/features/nodes/`
|
||||
|
||||
have a look at `state.nodes.invocation`
|
||||
|
||||
- on socket connect, if no schema saved, fetch `localhost:9090/openapi.json`, save JSON to `state.nodes.schema`
|
||||
- on fulfilled schema fetch, `parseSchema()` the schema. this outputs a `Record<string, Invocation>` which is saved to `state.nodes.invocations` - `Invocation` is like a template for the node
|
||||
- when you add a node, the the `Invocation` template is passed to `InvocationComponent.tsx` to build the UI component for that node
|
||||
- inputs/outputs have field types - and each field type gets an `FieldComponent` which includes a dispatcher to write state changes to redux `nodesSlice`
|
||||
- `reactflow` sends changes to nodes/edges to redux
|
||||
- to invoke, `buildNodesGraph()` state, then send this
|
||||
- changed onClick Invoke button actions to build the schema, then when schema builds it dispatches the actual network request to create the session - see `session.ts`
|
||||
29
invokeai/frontend/web/docs/PACKAGE_SCRIPTS.md
Normal file
29
invokeai/frontend/web/docs/PACKAGE_SCRIPTS.md
Normal file
@ -0,0 +1,29 @@
|
||||
# Package Scripts
|
||||
|
||||
WIP walkthrough of `package.json` scripts.
|
||||
|
||||
## `theme` & `theme:watch`
|
||||
|
||||
These run the Chakra CLI to generate types for the theme, or watch for code change and re-generate the types.
|
||||
|
||||
The CLI essentially monkeypatches Chakra's files in `node_modules`.
|
||||
|
||||
## `postinstall`
|
||||
|
||||
The `postinstall` script patches a few packages and runs the Chakra CLI to generate types for the theme.
|
||||
|
||||
### Patch `@chakra-ui/cli`
|
||||
|
||||
See: <https://github.com/chakra-ui/chakra-ui/issues/7394>
|
||||
|
||||
### Patch `redux-persist`
|
||||
|
||||
We want to persist the canvas state to `localStorage` but many canvas operations change data very quickly, so we need to debounce the writes to `localStorage`.
|
||||
|
||||
`redux-persist` is unfortunately unmaintained. The repo's current code is nonfunctional, but the last release's code depends on a package that was removed from `npm` for being malware, so we cannot just fork it.
|
||||
|
||||
So, we have to patch it directly. Perhaps a better way would be to write a debounced storage adapter, but I couldn't figure out how to do that.
|
||||
|
||||
### Patch `redux-deep-persist`
|
||||
|
||||
This package makes blacklisting and whitelisting persist configs very simple, but we have to patch it to match `redux-persist` for the types to work.
|
||||
49
invokeai/frontend/web/docs/README.md
Normal file
49
invokeai/frontend/web/docs/README.md
Normal file
@ -0,0 +1,49 @@
|
||||
# InvokeAI Web UI
|
||||
|
||||
- [InvokeAI Web UI](#invokeai-web-ui)
|
||||
- [Stack](#stack)
|
||||
- [Contributing](#contributing)
|
||||
- [Dev Environment](#dev-environment)
|
||||
- [Production builds](#production-builds)
|
||||
|
||||
The UI is a fairly straightforward Typescript React app. The only really fancy stuff is the Unified Canvas.
|
||||
|
||||
Code in `invokeai/frontend/web/` if you want to have a look.
|
||||
|
||||
## Stack
|
||||
|
||||
State management is Redux via [Redux Toolkit](https://github.com/reduxjs/redux-toolkit). Communication with server is a mix of HTTP and [socket.io](https://github.com/socketio/socket.io-client) (with a custom redux middleware to help).
|
||||
|
||||
[Chakra-UI](https://github.com/chakra-ui/chakra-ui) for components and styling.
|
||||
|
||||
[Konva](https://github.com/konvajs/react-konva) for the canvas, but we are pushing the limits of what is feasible with it (and HTML canvas in general). We plan to rebuild it with [PixiJS](https://github.com/pixijs/pixijs) to take advantage of WebGL's improved raster handling.
|
||||
|
||||
[Vite](https://vitejs.dev/) for bundling.
|
||||
|
||||
Localisation is via [i18next](https://github.com/i18next/react-i18next), but translation happens on our [Weblate](https://hosted.weblate.org/engage/invokeai/) project. Only the English source strings should be changed on this repo.
|
||||
|
||||
## Contributing
|
||||
|
||||
Thanks for your interest in contributing to the InvokeAI Web UI!
|
||||
|
||||
We encourage you to ping @psychedelicious and @blessedcoolant on [Discord](https://discord.gg/ZmtBAhwWhy) if you want to contribute, just to touch base and ensure your work doesn't conflict with anything else going on. The project is very active.
|
||||
|
||||
### Dev Environment
|
||||
|
||||
Install [node](https://nodejs.org/en/download/) and [yarn classic](https://classic.yarnpkg.com/lang/en/).
|
||||
|
||||
From `invokeai/frontend/web/` run `yarn install` to get everything set up.
|
||||
|
||||
Start everything in dev mode:
|
||||
|
||||
1. Start the dev server: `yarn dev`
|
||||
2. Start the InvokeAI UI per usual: `invokeai --web`
|
||||
3. Point your browser to the dev server address e.g. <http://localhost:5173/>
|
||||
|
||||
### Production builds
|
||||
|
||||
For a number of technical and logistical reasons, we need to commit UI build artefacts to the repo.
|
||||
|
||||
If you submit a PR, there is a good chance we will ask you to include a separate commit with a build of the app.
|
||||
|
||||
To build for production, run `yarn build`.
|
||||
Reference in New Issue
Block a user