From b793d69ff396ffa18e248f6ec23a7fd6cb6215d2 Mon Sep 17 00:00:00 2001 From: Matthias Wild <40327258+mauwii@users.noreply.github.com> Date: Sun, 16 Oct 2022 19:19:49 +0200 Subject: [PATCH] squash merge update-gh-actions into fix-gh-actions * fix mkdocs deployment * update path to python bin * add trigger for current branch * change path to python_bin for mac as well * try to use setup-python@v4 instead of setting env * remove setup conda action * try to use $CONDA * remove overseen action * change branch from master to main * sort out if then else for faster syntax * remove more if functions * add updates to create-caches as well * eliminate the rest of if functions * try to unpin pytorch and torchvision * restore pinned versions * try switching from set-output to use env * update test-invoke-conda as well * fix env var creation * quote variable * add second equal to compare * try another way to use outputs * fix outputs * pip install for mac before creating conda env * fix output variable * fix python bin path * remove pip install for before creating conda env * unpin streamlit version in conda mac env * try to make git-workflows better readable * remove 4gotten trigger * Update-gh-actions (#6) * fix mkdocs deployment * update path to python bin * add trigger for current branch * change path to python_bin for mac as well * try to use setup-python@v4 instead of setting env * remove setup conda action * try to use $CONDA * remove overseen action * change branch from master to main * sort out if then else for faster syntax * remove more if functions * add updates to create-caches as well * eliminate the rest of if functions * try to unpin pytorch and torchvision * restore pinned versions * try switching from set-output to use env * update test-invoke-conda as well * fix env var creation * quote variable * add second equal to compare * try another way to use outputs * fix outputs * pip install for mac before creating conda env * fix output variable * fix python bin path * remove pip install for before creating conda env * unpin streamlit version in conda mac env * try to make git-workflows better readable * use macos-latest * try to update conda before creating mac env * better conda update trial * re-pin streamlit version * re-added trigger to run workflow in current branch * try to find out if conda mac env could be updated * install cmake, protobuf and rust b4 conda * add yes to conda update * lets try anaconda3-2022.05 * try environment.yml for mac as well * reenable conda mac env, add pip install also fix gitignore by changing from dream to invoke * remove - unecesary virtualenv creation - conda update change != macos back to == linux * remove cmake from brew install since pre-installed * disable opencv-python pip requirement * fixed commands to find latest package versions * update requirements for mac env * back to the roots - only install conda env depending on runner_os with or without extra env variables * check out macOS in azure-pipelines since becoming kind of tired of the GitHub Runner which is broken as ... * let's try to setup python and update conda env * initialize conda before using it * add trigger in azure-pipelines.yml * And another go for update first .... * update azure-pipelines.yml - add caching - add checkpoint download - add paths to trigger and more * unquote checkpoint-url * fix chekpoint-url variable * mkdir before downloading model * set pr trigger to main, rename anaconda cache * unique cacheHitVariables * try to use macos-latest instead of macos-12 * update test-invoke-conda.yml: - remove unecesarry echo step - use s-weigand/setup-conda@v1 - remove conda update from install deps step since updated with action * update test-invoke-conda.yml: - rename conda env cache from ldm to invokeai - reorder steps: 1. checkout sources 2. setup python 3. setup conda 4. keep order after set platform variables * change macos back to 12 since also fails with 11 * update condition in run the tests make difference between main or not main * fix path to cache invokeai conda env * fix invokeai conda env cache path * update mkdocs-flow.yml * change conda-channel priority * update create-caches * update conda env also when cache was used * os dependend conda env cache path * use existing CONDA env pointing to conda root * create CONDA_ROOT output from $CONDA * use output variable to define test prompts * use setup-python v4, get rid of PYTHON_BIN env * add runner.os to result artifacts name * update test-invoke-conda.yml: - reuse macos-latest - disable setup python 3.9 - setup conda with default python version - create or update conda environment depending on cache success - remove name parameter from conda update since name is set in env yml * improve mkdocs-flow.yml * disable cache-hugginface-torch since preload_models.py downloads to more than one location * update mkdocs-flow.yml with new name * rename mkdocs action to mkdocs-material * try to ignore error when creating conda env maybe it would still be usable, lets see ;P * remove bloat * update environment-mac.yml to match dependencies of invoke-ai/InvokeAI's main branch * disable conda update, tweak prompt condition * try to set some env vars for macOS to fix conda * stop ignoring error, use env instead of outputs * tweak `[[` connditions * update python and pip dependencie makes a difference of 1 sec per itteration compared to 3.9!!! also I see no reason why using a old pip version would be beneficial * remove unecesarry env for macOS everything was pre-tested on my MacBook Air 2020 with M1 * update conda env in setup step * activate conda env after installation * update test-invoke-conda.yml - set conda env dependent on matrix.os - set CONDA_ENV_NAME to prevent breaking action when renaming conda env - fix conda env activation * fix activate conda env * set bash -l as default shell * use action to activate conda env * add conda env file to env activation * try to replace s-weigeand with conda-incubator * remove azure-pipelines.yml funniest part is that the macos runner is the same as the one on github! * include environment-file in matrix - also disable auto-activate-base and auto-update-conda - include macos-latest and macos-12 for debugging purpose - set miniforge-version in matrix * fix miniforge-variant, set fail-fast to false * add step to setup miniconda - make default shell a matrix variable - remove bloat * use a mac env yml without pinned versions * unpin nomkl, pytorch and torchvision also removed opencv-pyhton * cache conda pkgs dir instead of conda env * use python 3.10, exclude macos-12 from cache * fix expression * prepare for PR * fix doubled id * reuse pinned versions in mac conda env - updated python pip version - unpined pytorch and torchvision - removed opencv-python - updated versions to most recent (tested locally) * fix classical copy/paste error * remove unused env from shell-block comment * fix hashFiles function to determine restore-keys * reenable caching `~.cache`, update create-caches * unpin all versions in mac conda env file this was the only way I got it working in the action, also works locally tested on MacBook Air 2020 M1 remove environment-mac-unpinned.yml * prepare merge by removing this branch from trigger * include pull_request trigger for main and dev * remove pull_request trigger --- .github/workflows/create-caches.yml | 117 ++++++++++++++++-------- .github/workflows/mkdocs-flow.yml | 28 ------ .github/workflows/mkdocs-material.yml | 44 +++++++++ .github/workflows/test-invoke-conda.yml | 115 +++++++++++++---------- .gitignore | 4 +- environment-mac.yml | 82 ++++++++--------- 6 files changed, 228 insertions(+), 162 deletions(-) delete mode 100644 .github/workflows/mkdocs-flow.yml create mode 100644 .github/workflows/mkdocs-material.yml diff --git a/.github/workflows/create-caches.yml b/.github/workflows/create-caches.yml index e21286a407..0e0531eaa5 100644 --- a/.github/workflows/create-caches.yml +++ b/.github/workflows/create-caches.yml @@ -1,26 +1,43 @@ name: Create Caches -on: - workflow_dispatch + +on: workflow_dispatch + jobs: - build: + os_matrix: strategy: matrix: - os: [ ubuntu-latest, macos-12 ] - name: Create Caches on ${{ matrix.os }} conda + os: [ubuntu-latest, macos-latest] + include: + - os: ubuntu-latest + environment-file: environment.yml + default-shell: bash -l {0} + - os: macos-latest + environment-file: environment-mac.yml + default-shell: bash -l {0} + name: Test invoke.py on ${{ matrix.os }} with conda runs-on: ${{ matrix.os }} + defaults: + run: + shell: ${{ matrix.default-shell }} steps: - - name: Set platform variables - id: vars - run: | - if [ "$RUNNER_OS" = "macOS" ]; then - echo "::set-output name=ENV_FILE::environment-mac.yml" - echo "::set-output name=PYTHON_BIN::/usr/local/miniconda/envs/ldm/bin/python" - elif [ "$RUNNER_OS" = "Linux" ]; then - echo "::set-output name=ENV_FILE::environment.yml" - echo "::set-output name=PYTHON_BIN::/usr/share/miniconda/envs/ldm/bin/python" - fi - name: Checkout sources uses: actions/checkout@v3 + + - name: setup miniconda + uses: conda-incubator/setup-miniconda@v2 + with: + auto-activate-base: false + auto-update-conda: false + miniconda-version: latest + + - name: set environment + run: | + [[ "$GITHUB_REF" == 'refs/heads/main' ]] \ + && echo "TEST_PROMPTS=tests/preflight_prompts.txt" >> $GITHUB_ENV \ + || echo "TEST_PROMPTS=tests/dev_prompts.txt" >> $GITHUB_ENV + echo "CONDA_ROOT=$CONDA" >> $GITHUB_ENV + echo "CONDA_ENV_NAME=invokeai" >> $GITHUB_ENV + - name: Use Cached Stable Diffusion v1.4 Model id: cache-sd-v1-4 uses: actions/cache@v3 @@ -29,42 +46,62 @@ jobs: with: path: models/ldm/stable-diffusion-v1/model.ckpt key: ${{ env.cache-name }} - restore-keys: | - ${{ env.cache-name }} + restore-keys: ${{ env.cache-name }} + + - name: Use Cached Stable Diffusion v1.4 Model + id: cache-sd-v1-4 + uses: actions/cache@v3 + env: + cache-name: cache-sd-v1-4 + with: + path: models/ldm/stable-diffusion-v1/model.ckpt + key: ${{ env.cache-name }} + restore-keys: ${{ env.cache-name }} + - name: Download Stable Diffusion v1.4 Model if: ${{ steps.cache-sd-v1-4.outputs.cache-hit != 'true' }} run: | - if [ ! -e models/ldm/stable-diffusion-v1 ]; then - mkdir -p models/ldm/stable-diffusion-v1 - fi - if [ ! -e models/ldm/stable-diffusion-v1/model.ckpt ]; then - curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }} - fi - - name: Use Cached Dependencies - id: cache-conda-env-ldm + [[ -d models/ldm/stable-diffusion-v1 ]] \ + || mkdir -p models/ldm/stable-diffusion-v1 + [[ -r models/ldm/stable-diffusion-v1/model.ckpt ]] \ + || curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }} + + - name: Use cached Conda Environment uses: actions/cache@v3 env: - cache-name: cache-conda-env-ldm + cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }} + conda-env-file: ${{ matrix.environment-file }} with: - path: ~/.conda/envs/ldm + path: ${{ env.CONDA_ROOT }}/envs/${{ env.CONDA_ENV_NAME }} key: ${{ env.cache-name }} - restore-keys: | - ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(steps.vars.outputs.ENV_FILE) }} - - name: Install Dependencies - if: ${{ steps.cache-conda-env-ldm.outputs.cache-hit != 'true' }} - run: | - conda env create -f ${{ steps.vars.outputs.ENV_FILE }} - - name: Use Cached Huggingface and Torch models - id: cache-huggingface-torch + restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }} + + - name: Use cached Conda Packages uses: actions/cache@v3 env: - cache-name: cache-huggingface-torch + cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }} + conda-env-file: ${{ matrix.environment-file }} + with: + path: ${{ env.CONDA_PKGS_DIR }} + key: ${{ env.cache-name }} + restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }} + + - name: Activate Conda Env + uses: conda-incubator/setup-miniconda@v2 + with: + activate-environment: ${{ env.CONDA_ENV_NAME }} + environment-file: ${{ matrix.environment-file }} + + - name: Use Cached Huggingface and Torch models + id: cache-hugginface-torch + uses: actions/cache@v3 + env: + cache-name: cache-hugginface-torch with: path: ~/.cache key: ${{ env.cache-name }} restore-keys: | ${{ env.cache-name }}-${{ hashFiles('scripts/preload_models.py') }} - - name: Download Huggingface and Torch models - if: ${{ steps.cache-huggingface-torch.outputs.cache-hit != 'true' }} - run: | - ${{ steps.vars.outputs.PYTHON_BIN }} scripts/preload_models.py + + - name: run preload_models.py + run: python scripts/preload_models.py diff --git a/.github/workflows/mkdocs-flow.yml b/.github/workflows/mkdocs-flow.yml deleted file mode 100644 index 318f180c1b..0000000000 --- a/.github/workflows/mkdocs-flow.yml +++ /dev/null @@ -1,28 +0,0 @@ -name: Deploy -on: - push: - branches: - - main - # pull_request: - # branches: - # - main -jobs: - build: - name: Deploy docs to GitHub Pages - runs-on: ubuntu-latest - steps: - - name: Checkout - uses: actions/checkout@v3 - with: - fetch-depth: 0 - - name: Build - uses: Tiryoh/actions-mkdocs@v0 - with: - mkdocs_version: 'latest' # option - requirements: '/requirements-mkdocs.txt' # option - configfile: '/mkdocs.yml' # option - - name: Deploy - uses: peaceiris/actions-gh-pages@v3 - with: - github_token: ${{ secrets.GITHUB_TOKEN }} - publish_dir: ./site diff --git a/.github/workflows/mkdocs-material.yml b/.github/workflows/mkdocs-material.yml new file mode 100644 index 0000000000..2d01dec6df --- /dev/null +++ b/.github/workflows/mkdocs-material.yml @@ -0,0 +1,44 @@ +name: mkdocs-material +on: + push: + branches: + - 'main' + - 'development' + pull_request: + branches: + - 'main' + - 'development' + +jobs: + mkdocs-material: + runs-on: ubuntu-latest + steps: + - name: checkout sources + uses: actions/checkout@v3 + with: + fetch-depth: 0 + + - name: setup python + uses: actions/setup-python@v4 + with: + python-version: '3.10' + + - name: install requirements + run: | + python -m \ + pip install -r requirements-mkdocs.txt + + - name: confirm buildability + run: | + python -m \ + mkdocs build \ + --clean \ + --verbose + + - name: deploy to gh-pages + if: ${{ github.ref == 'refs/heads/main' }} + run: | + python -m \ + mkdocs gh-deploy \ + --clean \ + --force diff --git a/.github/workflows/test-invoke-conda.yml b/.github/workflows/test-invoke-conda.yml index b5314cfd6b..027d1bce47 100644 --- a/.github/workflows/test-invoke-conda.yml +++ b/.github/workflows/test-invoke-conda.yml @@ -4,29 +4,43 @@ on: branches: - 'main' - 'development' + jobs: os_matrix: strategy: matrix: - os: [ ubuntu-latest, macos-12 ] + os: [ubuntu-latest, macos-latest] + include: + - os: ubuntu-latest + environment-file: environment.yml + default-shell: bash -l {0} + - os: macos-latest + environment-file: environment-mac.yml + default-shell: bash -l {0} name: Test invoke.py on ${{ matrix.os }} with conda runs-on: ${{ matrix.os }} + defaults: + run: + shell: ${{ matrix.default-shell }} steps: - - run: | - echo The PR was merged - - name: Set platform variables - id: vars - run: | - # Note, can't "activate" via github action; specifying the env's python has the same effect - if [ "$RUNNER_OS" = "macOS" ]; then - echo "::set-output name=ENV_FILE::environment-mac.yml" - echo "::set-output name=PYTHON_BIN::/usr/local/miniconda/envs/ldm/bin/python" - elif [ "$RUNNER_OS" = "Linux" ]; then - echo "::set-output name=ENV_FILE::environment.yml" - echo "::set-output name=PYTHON_BIN::/usr/share/miniconda/envs/ldm/bin/python" - fi - name: Checkout sources uses: actions/checkout@v3 + + - name: setup miniconda + uses: conda-incubator/setup-miniconda@v2 + with: + auto-activate-base: false + auto-update-conda: false + miniconda-version: latest + + - name: set environment + run: | + [[ "$GITHUB_REF" == 'refs/heads/main' ]] \ + && echo "TEST_PROMPTS=tests/preflight_prompts.txt" >> $GITHUB_ENV \ + || echo "TEST_PROMPTS=tests/dev_prompts.txt" >> $GITHUB_ENV + echo "CONDA_ROOT=$CONDA" >> $GITHUB_ENV + echo "CONDA_ENV_NAME=invokeai" >> $GITHUB_ENV + - name: Use Cached Stable Diffusion v1.4 Model id: cache-sd-v1-4 uses: actions/cache@v3 @@ -35,31 +49,42 @@ jobs: with: path: models/ldm/stable-diffusion-v1/model.ckpt key: ${{ env.cache-name }} - restore-keys: | - ${{ env.cache-name }} + restore-keys: ${{ env.cache-name }} + - name: Download Stable Diffusion v1.4 Model if: ${{ steps.cache-sd-v1-4.outputs.cache-hit != 'true' }} run: | - if [ ! -e models/ldm/stable-diffusion-v1 ]; then - mkdir -p models/ldm/stable-diffusion-v1 - fi - if [ ! -e models/ldm/stable-diffusion-v1/model.ckpt ]; then - curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }} - fi - - name: Use Cached Dependencies - id: cache-conda-env-ldm + [[ -d models/ldm/stable-diffusion-v1 ]] \ + || mkdir -p models/ldm/stable-diffusion-v1 + [[ -r models/ldm/stable-diffusion-v1/model.ckpt ]] \ + || curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }} + + - name: Use cached Conda Environment uses: actions/cache@v3 env: - cache-name: cache-conda-env-ldm + cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }} + conda-env-file: ${{ matrix.environment-file }} with: - path: ~/.conda/envs/ldm + path: ${{ env.CONDA_ROOT }}/envs/${{ env.CONDA_ENV_NAME }} key: ${{ env.cache-name }} - restore-keys: | - ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(steps.vars.outputs.ENV_FILE) }} - - name: Install Dependencies - if: ${{ steps.cache-conda-env-ldm.outputs.cache-hit != 'true' }} - run: | - conda env create -f ${{ steps.vars.outputs.ENV_FILE }} + restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }} + + - name: Use cached Conda Packages + uses: actions/cache@v3 + env: + cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }} + conda-env-file: ${{ matrix.environment-file }} + with: + path: ${{ env.CONDA_PKGS_DIR }} + key: ${{ env.cache-name }} + restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }} + + - name: Activate Conda Env + uses: conda-incubator/setup-miniconda@v2 + with: + activate-environment: ${{ env.CONDA_ENV_NAME }} + environment-file: ${{ matrix.environment-file }} + - name: Use Cached Huggingface and Torch models id: cache-hugginface-torch uses: actions/cache@v3 @@ -70,28 +95,18 @@ jobs: key: ${{ env.cache-name }} restore-keys: | ${{ env.cache-name }}-${{ hashFiles('scripts/preload_models.py') }} - - name: Download Huggingface and Torch models - if: ${{ steps.cache-hugginface-torch.outputs.cache-hit != 'true' }} - run: | - ${{ steps.vars.outputs.PYTHON_BIN }} scripts/preload_models.py -# - name: Run tmate -# uses: mxschmitt/action-tmate@v3 -# timeout-minutes: 30 + + - name: run preload_models.py + run: python scripts/preload_models.py + - name: Run the tests run: | - # Note, can't "activate" via github action; specifying the env's python has the same effect - if [ $(uname) = "Darwin" ]; then - export PYTORCH_ENABLE_MPS_FALLBACK=1 - fi - # Utterly hacky, but I don't know how else to do this - if [[ ${{ github.ref }} == 'refs/heads/master' ]]; then - time ${{ steps.vars.outputs.PYTHON_BIN }} scripts/invoke.py --from_file tests/preflight_prompts.txt - elif [[ ${{ github.ref }} == 'refs/heads/development' ]]; then - time ${{ steps.vars.outputs.PYTHON_BIN }} scripts/invoke.py --from_file tests/dev_prompts.txt - fi + time python scripts/invoke.py \ + --from_file ${{ env.TEST_PROMPTS }} mkdir -p outputs/img-samples + - name: Archive results uses: actions/upload-artifact@v3 with: - name: results + name: results_${{ matrix.os }} path: outputs/img-samples diff --git a/.gitignore b/.gitignore index 0c3bd66d95..744ca1f1bf 100644 --- a/.gitignore +++ b/.gitignore @@ -1,7 +1,7 @@ # ignore default image save location and model symbolic link outputs/ models/ldm/stable-diffusion-v1/model.ckpt -ldm/dream/restoration/codeformer/weights +ldm/invoke/restoration/codeformer/weights # ignore the Anaconda/Miniconda installer used while building Docker image anaconda.sh @@ -180,7 +180,7 @@ src **/__pycache__/ outputs -# Logs and associated folders +# Logs and associated folders # created from generated embeddings. logs testtube diff --git a/environment-mac.yml b/environment-mac.yml index c71e8e67e1..735343c508 100644 --- a/environment-mac.yml +++ b/environment-mac.yml @@ -3,56 +3,54 @@ channels: - pytorch - conda-forge dependencies: - - python==3.9.13 - - pip==22.2.2 + - python>=3.10, <3.11 + - pip>=22 # pytorch left unpinned - - pytorch==1.12.1 - - torchvision==0.13.1 - + - pytorch + - torchvision # I suggest to keep the other deps sorted for convenience. # To determine what the latest versions should be, run: # # ```shell - # sed -E 's/ldm/ldm-updated/;20,99s/- ([^=]+)==.+/- \1/' environment-mac.yml > environment-mac-updated.yml - # CONDA_SUBDIR=osx-arm64 conda env create -f environment-mac-updated.yml && conda list -n ldm-updated | awk ' {print " - " $1 "==" $2;} ' + # sed -E 's/invokeai/invokeai-updated/;20,99s/- ([^=]+)==.+/- \1/' environment-mac.yml > environment-mac-updated.yml + # CONDA_SUBDIR=osx-arm64 conda env create -f environment-mac-updated.yml && conda list -n invokeai-updated | awk ' {print " - " $1 "==" $2;} ' # ``` - - albumentations==1.2.1 - - coloredlogs==15.0.1 - - einops==0.4.1 - - grpcio==1.46.4 - - humanfriendly==10.0 - - imageio==2.21.2 - - imageio-ffmpeg==0.4.7 - - imgaug==0.4.0 - - kornia==0.6.7 - - mpmath==1.2.1 - - nomkl=1.0 - - numpy==1.23.2 - - omegaconf==2.1.1 - - openh264==2.3.0 - - onnx==1.12.0 - - onnxruntime==1.12.1 - - pudb==2022.1 - - pytorch-lightning==1.7.5 - - scipy==1.9.1 - - streamlit==1.12.2 - - sympy==1.10.1 - - tensorboard==2.10.0 - - torchmetrics==0.9.3 + - albumentations + - coloredlogs + - einops + - grpcio + - humanfriendly + - imageio + - imageio-ffmpeg + - imgaug + - kornia + - mpmath + - nomkl + - numpy + - omegaconf + - openh264 + - onnx + - onnxruntime + - pudb + - pytorch-lightning + - scipy + - streamlit + - sympy + - tensorboard + - torchmetrics - pip: - - flask==2.1.3 - - flask_socketio==5.3.0 - - flask_cors==3.0.10 - - dependency_injector==4.40.0 - - eventlet==0.33.1 - - opencv-python==4.6.0 - - protobuf==3.19.5 - - realesrgan==0.2.5.0 - - send2trash==1.8.0 - - test-tube==0.7.5 - - transformers==4.21.2 - - torch-fidelity==0.3.0 + - flask + - flask_socketio + - flask_cors + - dependency_injector + - eventlet + - protobuf + - realesrgan + - send2trash + - test-tube + - transformers + - torch-fidelity - -e git+https://github.com/CompVis/taming-transformers.git@master#egg=taming-transformers - -e git+https://github.com/openai/CLIP.git@main#egg=clip - -e git+https://github.com/Birch-san/k-diffusion.git@mps#egg=k_diffusion