Mirrored_Repos/InvokeAI
1
0
Fork 0
mirror of https://github.com/invoke-ai/InvokeAI synced 2024-08-30 20:32:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

squash merge update-gh-actions into fix-gh-actions

Browse Source 
* fix mkdocs deployment

* update path to python bin

* add trigger for current branch

* change path to python_bin for mac as well

* try to use setup-python@v4 instead of setting env

* remove setup conda action

* try to use $CONDA

* remove overseen action

* change branch from master to main

* sort out if then else for faster syntax

* remove more if functions

* add updates to create-caches as well

* eliminate the rest of if functions

* try to unpin pytorch and torchvision

* restore pinned versions

* try switching from set-output to use env

* update test-invoke-conda as well

* fix env var creation

* quote variable

* add second equal to compare

* try another way to use outputs

* fix outputs

* pip install for mac before creating conda env

* fix output variable

* fix python bin path

* remove pip install for before creating conda env

* unpin streamlit version in conda mac env

* try to make git-workflows better readable

* remove 4gotten trigger

* Update-gh-actions (#6)

* fix mkdocs deployment

* update path to python bin

* add trigger for current branch

* change path to python_bin for mac as well

* try to use setup-python@v4 instead of setting env

* remove setup conda action

* try to use $CONDA

* remove overseen action

* change branch from master to main

* sort out if then else for faster syntax

* remove more if functions

* add updates to create-caches as well

* eliminate the rest of if functions

* try to unpin pytorch and torchvision

* restore pinned versions

* try switching from set-output to use env

* update test-invoke-conda as well

* fix env var creation

* quote variable

* add second equal to compare

* try another way to use outputs

* fix outputs

* pip install for mac before creating conda env

* fix output variable

* fix python bin path

* remove pip install for before creating conda env

* unpin streamlit version in conda mac env

* try to make git-workflows better readable

* use macos-latest

* try to update conda before creating mac env

* better conda update trial

* re-pin streamlit version

* re-added trigger to run workflow in current branch

* try to find out if conda mac env could be updated

* install cmake, protobuf and rust b4 conda

* add yes to conda update

* lets try anaconda3-2022.05

* try environment.yml for mac as well

* reenable conda mac env, add pip install
also fix gitignore by changing from dream to invoke

* remove
- unecesary virtualenv creation
- conda update

change != macos back to == linux

* remove cmake from brew install since pre-installed

* disable opencv-python pip requirement

* fixed commands to find latest package versions

* update requirements for mac env

* back to the roots - only install conda env
depending on runner_os with or without extra env variables

* check out macOS in azure-pipelines
since becoming kind of tired of the GitHub Runner which is broken as ...

* let's try to setup python and update conda env

* initialize conda before using it

* add trigger in azure-pipelines.yml

* And another go for update first ....

* update azure-pipelines.yml
- add caching
- add checkpoint download
- add paths to trigger
and more

* unquote checkpoint-url

* fix chekpoint-url variable

* mkdir before downloading model

* set pr trigger to main, rename anaconda cache

* unique cacheHitVariables

* try to use macos-latest instead of macos-12

* update test-invoke-conda.yml:
- remove unecesarry echo step
- use s-weigand/setup-conda@v1
- remove conda update from install deps step since updated with action

* update test-invoke-conda.yml:
- rename conda env cache from ldm to invokeai
- reorder steps:
  1. checkout sources
  2. setup python
  3. setup conda
  4. keep order after set platform variables

* change macos back to 12 since also fails with 11

* update condition in run the tests
make difference between main or not main

* fix path to cache invokeai conda env

* fix invokeai conda env cache path

* update mkdocs-flow.yml

* change conda-channel priority

* update create-caches

* update conda env also when cache was used

* os dependend conda env cache path

* use existing CONDA env pointing to conda root

* create CONDA_ROOT output from $CONDA

* use output variable to define test prompts

* use setup-python v4, get rid of PYTHON_BIN env

* add runner.os to result artifacts name

* update test-invoke-conda.yml:
- reuse macos-latest
- disable setup python 3.9
- setup conda with default python version
- create or update conda environment depending on cache success
- remove name parameter from conda update since name is set in env yml

* improve mkdocs-flow.yml

* disable cache-hugginface-torch
since preload_models.py downloads to more than one location

* update mkdocs-flow.yml with new name

* rename mkdocs action to mkdocs-material

* try to ignore error when creating conda env
maybe it would still be usable, lets see ;P

* remove bloat

* update environment-mac.yml
to match dependencies of invoke-ai/InvokeAI's main branch

* disable conda update, tweak prompt condition

* try to set some env vars for macOS to fix conda

* stop ignoring error, use env instead of outputs

* tweak `[[` connditions

* update python and pip dependencie
makes a difference of 1 sec per itteration compared to 3.9!!!
also I see no reason why using a old pip version would be beneficial

* remove unecesarry env for macOS
everything was pre-tested on my MacBook Air 2020 with M1

* update conda env in setup step

* activate conda env after installation

* update test-invoke-conda.yml
- set conda env dependent on matrix.os
- set CONDA_ENV_NAME to prevent breaking action when renaming conda env
- fix conda env activation

* fix activate conda env

* set bash -l as default shell

* use action to activate conda env

* add conda env file to env activation

* try to replace s-weigeand with conda-incubator

* remove azure-pipelines.yml
funniest part is that the macos runner is the same as the one on github!

* include environment-file in matrix
- also disable auto-activate-base and auto-update-conda
- include macos-latest and macos-12 for debugging purpose
- set miniforge-version in matrix

* fix miniforge-variant, set fail-fast to false

* add step to setup miniconda
- make default shell a matrix variable
- remove bloat

* use a mac env yml without pinned versions

* unpin nomkl, pytorch and torchvision
also removed opencv-pyhton

* cache conda pkgs dir instead of conda env

* use python 3.10, exclude macos-12 from cache

* fix expression

* prepare for PR

* fix doubled id

* reuse pinned versions in mac conda env
- updated python pip version
- unpined pytorch and torchvision
- removed opencv-python
- updated versions to most recent (tested locally)

* fix classical copy/paste error

* remove unused env from shell-block comment

* fix hashFiles function to determine restore-keys

* reenable caching `~.cache`, update create-caches

* unpin all versions in mac conda env file
this was the only way I got it working in the action, also works locally
tested on MacBook Air 2020 M1
remove environment-mac-unpinned.yml

* prepare merge by removing this branch from trigger

* include pull_request trigger for main and dev

* remove pull_request trigger
This commit is contained in:
Matthias Wild 2022-10-16 19:19:49 +02:00 committed by mauwii
parent a8b794d7e0
commit 44ff8f8531
No known key found for this signature in database
GPG Key ID: D923DB04ADB3F5AB
6 changed files with 227 additions and 161 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

117
.github/workflows/create-caches.yml vendored
View File

@ -1,26 +1,43 @@
name: Create Caches
on:
workflow_dispatch
on: workflow_dispatch
jobs:
build:
os_matrix:
strategy:
matrix:
os: [ ubuntu-latest, macos-12 ]
name: Create Caches on ${{ matrix.os }} conda
os: [ubuntu-latest, macos-latest]
include:
- os: ubuntu-latest
environment-file: environment.yml
default-shell: bash -l {0}
- os: macos-latest
environment-file: environment-mac.yml
default-shell: bash -l {0}
name: Test invoke.py on ${{ matrix.os }} with conda
runs-on: ${{ matrix.os }}
defaults:
run:
shell: ${{ matrix.default-shell }}
steps:
- name: Set platform variables
id: vars
run: |
if [ "$RUNNER_OS" = "macOS" ]; then
echo "::set-output name=ENV_FILE::environment-mac.yml"
echo "::set-output name=PYTHON_BIN::/usr/local/miniconda/envs/ldm/bin/python"
elif [ "$RUNNER_OS" = "Linux" ]; then
echo "::set-output name=ENV_FILE::environment.yml"
echo "::set-output name=PYTHON_BIN::/usr/share/miniconda/envs/ldm/bin/python"
fi
- name: Checkout sources
uses: actions/checkout@v3
- name: setup miniconda
uses: conda-incubator/setup-miniconda@v2
with:
auto-activate-base: false
auto-update-conda: false
miniconda-version: latest
- name: set environment
run: |
[[ "$GITHUB_REF" == 'refs/heads/main' ]] \
&& echo "TEST_PROMPTS=tests/preflight_prompts.txt" >> $GITHUB_ENV \
|| echo "TEST_PROMPTS=tests/dev_prompts.txt" >> $GITHUB_ENV
echo "CONDA_ROOT=$CONDA" >> $GITHUB_ENV
echo "CONDA_ENV_NAME=invokeai" >> $GITHUB_ENV
- name: Use Cached Stable Diffusion v1.4 Model
id: cache-sd-v1-4
uses: actions/cache@v3
@ -29,42 +46,62 @@ jobs:
with:
path: models/ldm/stable-diffusion-v1/model.ckpt
key: ${{ env.cache-name }}
restore-keys: |
${{ env.cache-name }}
restore-keys: ${{ env.cache-name }}
- name: Use Cached Stable Diffusion v1.4 Model
id: cache-sd-v1-4
uses: actions/cache@v3
env:
cache-name: cache-sd-v1-4
with:
path: models/ldm/stable-diffusion-v1/model.ckpt
key: ${{ env.cache-name }}
restore-keys: ${{ env.cache-name }}
- name: Download Stable Diffusion v1.4 Model
if: ${{ steps.cache-sd-v1-4.outputs.cache-hit != 'true' }}
run: |
if [ ! -e models/ldm/stable-diffusion-v1 ]; then
mkdir -p models/ldm/stable-diffusion-v1
fi
if [ ! -e models/ldm/stable-diffusion-v1/model.ckpt ]; then
curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }}
fi
- name: Use Cached Dependencies
id: cache-conda-env-ldm
[[ -d models/ldm/stable-diffusion-v1 ]] \
|| mkdir -p models/ldm/stable-diffusion-v1
[[ -r models/ldm/stable-diffusion-v1/model.ckpt ]] \
|| curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }}
- name: Use cached Conda Environment
uses: actions/cache@v3
env:
cache-name: cache-conda-env-ldm
cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }}
conda-env-file: ${{ matrix.environment-file }}
with:
path: ~/.conda/envs/ldm
path: ${{ env.CONDA_ROOT }}/envs/${{ env.CONDA_ENV_NAME }}
key: ${{ env.cache-name }}
restore-keys: |
${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(steps.vars.outputs.ENV_FILE) }}
- name: Install Dependencies
if: ${{ steps.cache-conda-env-ldm.outputs.cache-hit != 'true' }}
run: |
conda env create -f ${{ steps.vars.outputs.ENV_FILE }}
- name: Use Cached Huggingface and Torch models
id: cache-huggingface-torch
restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }}
- name: Use cached Conda Packages
uses: actions/cache@v3
env:
cache-name: cache-huggingface-torch
cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }}
conda-env-file: ${{ matrix.environment-file }}
with:
path: ${{ env.CONDA_PKGS_DIR }}
key: ${{ env.cache-name }}
restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }}
- name: Activate Conda Env
uses: conda-incubator/setup-miniconda@v2
with:
activate-environment: ${{ env.CONDA_ENV_NAME }}
environment-file: ${{ matrix.environment-file }}
- name: Use Cached Huggingface and Torch models
id: cache-hugginface-torch
uses: actions/cache@v3
env:
cache-name: cache-hugginface-torch
with:
path: ~/.cache
key: ${{ env.cache-name }}
restore-keys: |
${{ env.cache-name }}-${{ hashFiles('scripts/preload_models.py') }}
- name: Download Huggingface and Torch models
if: ${{ steps.cache-huggingface-torch.outputs.cache-hit != 'true' }}
run: |
${{ steps.vars.outputs.PYTHON_BIN }} scripts/preload_models.py
- name: run preload_models.py
run: python scripts/preload_models.py

28
.github/workflows/mkdocs-flow.yml vendored
View File

@ -1,28 +0,0 @@
name: Deploy
on:
push:
branches:
- main
# pull_request:
# branches:
# - main
jobs:
build:
name: Deploy docs to GitHub Pages
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Build
uses: Tiryoh/actions-mkdocs@v0
with:
mkdocs_version: 'latest' # option
requirements: '/requirements-mkdocs.txt' # option
configfile: '/mkdocs.yml' # option
- name: Deploy
uses: peaceiris/actions-gh-pages@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./site

44
.github/workflows/mkdocs-material.yml vendored Normal file
View File

@ -0,0 +1,44 @@
name: mkdocs-material
on:
push:
branches:
- 'main'
- 'development'
pull_request:
branches:
- 'main'
- 'development'
jobs:
mkdocs-material:
runs-on: ubuntu-latest
steps:
- name: checkout sources
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: setup python
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: install requirements
run: |
python -m \
pip install -r requirements-mkdocs.txt
- name: confirm buildability
run: |
python -m \
mkdocs build \
--clean \
--verbose
- name: deploy to gh-pages
if: ${{ github.ref == 'refs/heads/main' }}
run: |
python -m \
mkdocs gh-deploy \
--clean \
--force

115
.github/workflows/test-invoke-conda.yml vendored
View File

@ -4,29 +4,43 @@ on:
branches:
- 'main'
- 'development'
jobs:
os_matrix:
strategy:
matrix:
os: [ ubuntu-latest, macos-12 ]
os: [ubuntu-latest, macos-latest]
include:
- os: ubuntu-latest
environment-file: environment.yml
default-shell: bash -l {0}
- os: macos-latest
environment-file: environment-mac.yml
default-shell: bash -l {0}
name: Test invoke.py on ${{ matrix.os }} with conda
runs-on: ${{ matrix.os }}
defaults:
run:
shell: ${{ matrix.default-shell }}
steps:
- run: |
echo The PR was merged
- name: Set platform variables
id: vars
run: |
# Note, can't "activate" via github action; specifying the env's python has the same effect
if [ "$RUNNER_OS" = "macOS" ]; then
echo "::set-output name=ENV_FILE::environment-mac.yml"
echo "::set-output name=PYTHON_BIN::/usr/local/miniconda/envs/ldm/bin/python"
elif [ "$RUNNER_OS" = "Linux" ]; then
echo "::set-output name=ENV_FILE::environment.yml"
echo "::set-output name=PYTHON_BIN::/usr/share/miniconda/envs/ldm/bin/python"
fi
- name: Checkout sources
uses: actions/checkout@v3
- name: setup miniconda
uses: conda-incubator/setup-miniconda@v2
with:
auto-activate-base: false
auto-update-conda: false
miniconda-version: latest
- name: set environment
run: |
[[ "$GITHUB_REF" == 'refs/heads/main' ]] \
&& echo "TEST_PROMPTS=tests/preflight_prompts.txt" >> $GITHUB_ENV \
|| echo "TEST_PROMPTS=tests/dev_prompts.txt" >> $GITHUB_ENV
echo "CONDA_ROOT=$CONDA" >> $GITHUB_ENV
echo "CONDA_ENV_NAME=invokeai" >> $GITHUB_ENV
- name: Use Cached Stable Diffusion v1.4 Model
id: cache-sd-v1-4
uses: actions/cache@v3
@ -35,31 +49,42 @@ jobs:
with:
path: models/ldm/stable-diffusion-v1/model.ckpt
key: ${{ env.cache-name }}
restore-keys: |
${{ env.cache-name }}
restore-keys: ${{ env.cache-name }}
- name: Download Stable Diffusion v1.4 Model
if: ${{ steps.cache-sd-v1-4.outputs.cache-hit != 'true' }}
run: |
if [ ! -e models/ldm/stable-diffusion-v1 ]; then
mkdir -p models/ldm/stable-diffusion-v1
fi
if [ ! -e models/ldm/stable-diffusion-v1/model.ckpt ]; then
curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }}
fi
- name: Use Cached Dependencies
id: cache-conda-env-ldm
[[ -d models/ldm/stable-diffusion-v1 ]] \
|| mkdir -p models/ldm/stable-diffusion-v1
[[ -r models/ldm/stable-diffusion-v1/model.ckpt ]] \
|| curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }}
- name: Use cached Conda Environment
uses: actions/cache@v3
env:
cache-name: cache-conda-env-ldm
cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }}
conda-env-file: ${{ matrix.environment-file }}
with:
path: ~/.conda/envs/ldm
path: ${{ env.CONDA_ROOT }}/envs/${{ env.CONDA_ENV_NAME }}
key: ${{ env.cache-name }}
restore-keys: |
${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(steps.vars.outputs.ENV_FILE) }}
- name: Install Dependencies
if: ${{ steps.cache-conda-env-ldm.outputs.cache-hit != 'true' }}
run: |
conda env create -f ${{ steps.vars.outputs.ENV_FILE }}
restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }}
- name: Use cached Conda Packages
uses: actions/cache@v3
env:
cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }}
conda-env-file: ${{ matrix.environment-file }}
with:
path: ${{ env.CONDA_PKGS_DIR }}
key: ${{ env.cache-name }}
restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }}
- name: Activate Conda Env
uses: conda-incubator/setup-miniconda@v2
with:
activate-environment: ${{ env.CONDA_ENV_NAME }}
environment-file: ${{ matrix.environment-file }}
- name: Use Cached Huggingface and Torch models
id: cache-hugginface-torch
uses: actions/cache@v3
@ -70,28 +95,18 @@ jobs:
key: ${{ env.cache-name }}
restore-keys: |
${{ env.cache-name }}-${{ hashFiles('scripts/preload_models.py') }}
- name: Download Huggingface and Torch models
if: ${{ steps.cache-hugginface-torch.outputs.cache-hit != 'true' }}
run: |
${{ steps.vars.outputs.PYTHON_BIN }} scripts/preload_models.py
# - name: Run tmate
# uses: mxschmitt/action-tmate@v3
# timeout-minutes: 30
- name: run preload_models.py
run: python scripts/preload_models.py
- name: Run the tests
run: |
# Note, can't "activate" via github action; specifying the env's python has the same effect
if [ $(uname) = "Darwin" ]; then
export PYTORCH_ENABLE_MPS_FALLBACK=1
fi
# Utterly hacky, but I don't know how else to do this
if [[ ${{ github.ref }} == 'refs/heads/master' ]]; then
time ${{ steps.vars.outputs.PYTHON_BIN }} scripts/invoke.py --from_file tests/preflight_prompts.txt
elif [[ ${{ github.ref }} == 'refs/heads/development' ]]; then
time ${{ steps.vars.outputs.PYTHON_BIN }} scripts/invoke.py --from_file tests/dev_prompts.txt
fi
time python scripts/invoke.py \
--from_file ${{ env.TEST_PROMPTS }}
mkdir -p outputs/img-samples
- name: Archive results
uses: actions/upload-artifact@v3
with:
name: results
name: results_${{ matrix.os }}
path: outputs/img-samples

2
.gitignore vendored
View File

@ -180,7 +180,7 @@ src
**/__pycache__/
outputs
# Logs and associated folders
# Logs and associated folders
# created from generated embeddings.
logs
testtube

82
environment-mac.yml
View File

@ -3,56 +3,54 @@ channels:
- pytorch
- conda-forge
dependencies:
- python==3.9.13
- pip==22.2.2
- python>=3.10, <3.11
- pip>=22
# pytorch left unpinned
- pytorch==1.12.1
- torchvision==0.13.1
- pytorch
- torchvision
# I suggest to keep the other deps sorted for convenience.
# To determine what the latest versions should be, run:
#
# ```shell
# sed -E 's/ldm/ldm-updated/;20,99s/- ([^=]+)==.+/- \1/' environment-mac.yml > environment-mac-updated.yml
# CONDA_SUBDIR=osx-arm64 conda env create -f environment-mac-updated.yml && conda list -n ldm-updated | awk ' {print " - " $1 "==" $2;} '
# sed -E 's/invokeai/invokeai-updated/;20,99s/- ([^=]+)==.+/- \1/' environment-mac.yml > environment-mac-updated.yml
# CONDA_SUBDIR=osx-arm64 conda env create -f environment-mac-updated.yml && conda list -n invokeai-updated | awk ' {print " - " $1 "==" $2;} '
# ```
- albumentations==1.2.1
- coloredlogs==15.0.1
- einops==0.4.1
- grpcio==1.46.4
- humanfriendly==10.0
- imageio==2.21.2
- imageio-ffmpeg==0.4.7
- imgaug==0.4.0
- kornia==0.6.7
- mpmath==1.2.1
- nomkl=1.0
- numpy==1.23.2
- omegaconf==2.1.1
- openh264==2.3.0
- onnx==1.12.0
- onnxruntime==1.12.1
- pudb==2022.1
- pytorch-lightning==1.7.5
- scipy==1.9.1
- streamlit==1.12.2
- sympy==1.10.1
- tensorboard==2.10.0
- torchmetrics==0.9.3
- albumentations
- coloredlogs
- einops
- grpcio
- humanfriendly
- imageio
- imageio-ffmpeg
- imgaug
- kornia
- mpmath
- nomkl
- numpy
- omegaconf
- openh264
- onnx
- onnxruntime
- pudb
- pytorch-lightning
- scipy
- streamlit
- sympy
- tensorboard
- torchmetrics
- pip:
- flask==2.1.3
- flask_socketio==5.3.0
- flask_cors==3.0.10
- dependency_injector==4.40.0
- eventlet==0.33.1
- opencv-python==4.6.0
- protobuf==3.19.5
- realesrgan==0.2.5.0
- send2trash==1.8.0
- test-tube==0.7.5
- transformers==4.21.2
- torch-fidelity==0.3.0
- flask
- flask_socketio
- flask_cors
- dependency_injector
- eventlet
- protobuf
- realesrgan
- send2trash
- test-tube
- transformers
- torch-fidelity
- -e git+https://github.com/CompVis/taming-transformers.git@master#egg=taming-transformers
- -e git+https://github.com/openai/CLIP.git@main#egg=clip
- -e git+https://github.com/Birch-san/k-diffusion.git@mps#egg=k_diffusion