Mirrored_Repos/InvokeAI
1
0
Fork 0
mirror of https://github.com/invoke-ai/InvokeAI synced 2024-08-30 20:32:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

update (docker-)build scripts, .dockerignore and add patchmatch (#1970)

Browse Source 
* update build scripts and dockerignore
updates to build and run script:
- read repository name
- include flavor in container name
- read arch via arch command
- use latest tag instead of arch
- don't bindmount `$HOME/.huggingface`
- make sure HUGGINGFACE_TOKEN is set

updates to .dockerignore
- include environment-and-requirements
- exclude binary_installer
- exclude docker-build
- exclude docs

* disable push and pr triggers of cloud image
also disable pushing.

This was decided since:
- it is not multiarch useable
- the default image is already cloud aproved

* integrate patchmatch in container

* pin verisons of recently introduced dependencies

* remove now unecesarry part from build.sh
move huggingface token to run script, so it can download missing models

* move GPU_FLAGS to run script
since not needed at build time

* update env.sh

- read REPOSITORY_NAME from env if available
- add comment to explain the intension of this file
- remove unecesarry exports

* get rid of repository_name_lc

* capitalize variables

* update INSTALL_DOCKER with new variables

* add comments pointing to the docs

Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
This commit is contained in:
Matthias Wild 2022-12-16 13:53:37 +01:00 committed by GitHub
parent 7d09d9da49
commit c0c4d7ca69
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
7 changed files with 83 additions and 88 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
.dockerignore
View File

@ -1,16 +1,13 @@
* *
!backend !backend
!environments-and-requirements
!frontend !frontend
!binary_installer
!ldm !ldm
!main.py !main.py
!scripts !scripts
!server !server
!static !static
!setup.py !setup.py
!docker-build
!docs
docker-build/Dockerfile
# Guard against pulling in any models that might exist in the directory tree # Guard against pulling in any models that might exist in the directory tree
**/*.pt* **/*.pt*
@ -19,8 +16,4 @@ docker-build/Dockerfile
!configs !configs
configs/models.yaml configs/models.yaml
# unignore environment dirs/files, but ignore the environment.yml file or symlink in case it exists
!environment*
environment.yml
**/__pycache__ **/__pycache__

20
.github/workflows/build-cloud-img.yml vendored
View File

@ -1,15 +1,15 @@
name: Build and push cloud image name: Build and push cloud image
on: on:
workflow_dispatch: workflow_dispatch:
push: # push:
branches: # branches:
- main # - main
tags: # tags:
- v* # - v*
# we will NOT push the image on pull requests, only test buildability. # # we will NOT push the image on pull requests, only test buildability.
pull_request: # pull_request:
branches: # branches:
- main # - main
permissions: permissions:
contents: read contents: read
@ -82,6 +82,6 @@ jobs:
file: docker-build/Dockerfile.cloud file: docker-build/Dockerfile.cloud
platforms: Linux/${{ matrix.arch }} platforms: Linux/${{ matrix.arch }}
# do not push the image on PRs # do not push the image on PRs
push: ${{ github.event_name != 'pull_request' }} push: false
tags: ${{ steps.meta.outputs.tags }} tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }} labels: ${{ steps.meta.outputs.labels }}

27
docker-build/Dockerfile
View File

@ -14,9 +14,10 @@ RUN apt-get update \
&& apt-get clean \ && apt-get clean \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
# set workdir, PATH and copy sources # set WORKDIR, PATH and copy sources
WORKDIR /usr/src/app ARG WORKDIR=/usr/src/app
ENV PATH /usr/src/app/.venv/bin:$PATH WORKDIR ${WORKDIR}
ENV PATH ${WORKDIR}/.venv/bin:$PATH
ARG PIP_REQUIREMENTS=requirements-lin-cuda.txt ARG PIP_REQUIREMENTS=requirements-lin-cuda.txt
COPY . ./environments-and-requirements/${PIP_REQUIREMENTS} ./ COPY . ./environments-and-requirements/${PIP_REQUIREMENTS} ./
@ -38,18 +39,28 @@ FROM python:3.10-slim AS runtime
RUN apt-get update \ RUN apt-get update \
&& apt-get install -y \ && apt-get install -y \
--no-install-recommends \ --no-install-recommends \
build-essential=12.9 \
libgl1-mesa-glx=20.3.* \ libgl1-mesa-glx=20.3.* \
libglib2.0-0=2.66.* \ libglib2.0-0=2.66.* \
libopencv-dev=4.5.* \
python3-opencv=4.5.* \
&& apt-get clean \ && apt-get clean \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
WORKDIR /usr/src/app # setup environment
COPY --from=builder /usr/src/app . ARG WORKDIR=/usr/src/app
WORKDIR ${WORKDIR}
# set Environment, Entrypoint and default CMD COPY --from=builder ${WORKDIR} .
ENV PATH=${WORKDIR}/.venv/bin:$PATH
ENV INVOKEAI_ROOT /data ENV INVOKEAI_ROOT /data
ENV INVOKE_MODEL_RECONFIGURE --yes ENV INVOKE_MODEL_RECONFIGURE --yes
ENV PATH=/usr/src/app/.venv/bin:$PATH
# Initialize patchmatch
RUN ln -sf \
/usr/lib/"$(arch)"-linux-gnu/pkgconfig/opencv4.pc \
/usr/lib/"$(arch)"-linux-gnu/pkgconfig/opencv.pc \
&& python3 -c "from patchmatch import patch_match"
# set Entrypoint and default CMD
ENTRYPOINT [ "python3", "scripts/invoke.py" ] ENTRYPOINT [ "python3", "scripts/invoke.py" ]
CMD [ "--web", "--host=0.0.0.0" ] CMD [ "--web", "--host=0.0.0.0" ]

46
docker-build/build.sh
View File

@ -1,49 +1,35 @@
#!/usr/bin/env bash #!/usr/bin/env bash
set -e set -e
# IMPORTANT: You need to have a token on huggingface.co to be able to download the checkpoints!!! # How to use: https://invoke-ai.github.io/InvokeAI/installation/INSTALL_DOCKER/#setup
# configure values by using env when executing build.sh f.e. `env ARCH=aarch64 ./build.sh`
source ./docker-build/env.sh \ source ./docker-build/env.sh \
|| echo "please execute docker-build/build.sh from repository root" \ || echo "please execute docker-build/build.sh from repository root" \
|| exit 1 || exit 1
pip_requirements=${PIP_REQUIREMENTS:-requirements-lin-cuda.txt} PIP_REQUIREMENTS=${PIP_REQUIREMENTS:-requirements-lin-cuda.txt}
dockerfile=${INVOKE_DOCKERFILE:-docker-build/Dockerfile} DOCKERFILE=${INVOKE_DOCKERFILE:-docker-build/Dockerfile}
# print the settings # print the settings
echo -e "You are using these values:\n" echo -e "You are using these values:\n"
echo -e "Dockerfile:\t ${dockerfile}" echo -e "Dockerfile:\t ${DOCKERFILE}"
echo -e "requirements:\t ${pip_requirements}" echo -e "Requirements:\t ${PIP_REQUIREMENTS}"
echo -e "volumename:\t ${volumename}" echo -e "Volumename:\t ${VOLUMENAME}"
echo -e "arch:\t\t ${arch}" echo -e "arch:\t\t ${ARCH}"
echo -e "platform:\t ${platform}" echo -e "Platform:\t ${PLATFORM}"
echo -e "invokeai_tag:\t ${invokeai_tag}\n" echo -e "Invokeai_tag:\t ${INVOKEAI_TAG}\n"
if [[ -n "$(docker volume ls -f name="${volumename}" -q)" ]]; then if [[ -n "$(docker volume ls -f name="${VOLUMENAME}" -q)" ]]; then
echo "Volume already exists" echo -e "Volume already exists\n"
echo
else else
echo -n "createing docker volume " echo -n "createing docker volume "
docker volume create "${volumename}" docker volume create "${VOLUMENAME}"
fi fi
# Build Container # Build Container
docker build \ docker build \
--platform="${platform}" \ --platform="${PLATFORM}" \
--tag="${invokeai_tag}" \ --tag="${INVOKEAI_TAG}" \
--build-arg="PIP_REQUIREMENTS=${pip_requirements}" \ --build-arg="PIP_REQUIREMENTS=${PIP_REQUIREMENTS}" \
--file="${dockerfile}" \ --file="${DOCKERFILE}" \
. .
docker run \
--rm \
--platform="$platform" \
--name="$project_name" \
--hostname="$project_name" \
--mount="source=$volumename,target=/data" \
--mount="type=bind,source=$HOME/.huggingface,target=/root/.huggingface" \
--env="HUGGINGFACE_TOKEN=${HUGGINGFACE_TOKEN}" \
--entrypoint="python3" \
"${invokeai_tag}" \
scripts/configure_invokeai.py --yes

20
docker-build/env.sh
View File

@ -1,15 +1,9 @@
#!/usr/bin/env bash #!/usr/bin/env bash
project_name=${PROJECT_NAME:-invokeai} # Variables shared by build.sh and run.sh
volumename=${VOLUMENAME:-${project_name}_data} REPOSITORY_NAME=${REPOSITORY_NAME:-$(basename "$(git rev-parse --show-toplevel)")}
arch=${ARCH:-x86_64} VOLUMENAME=${VOLUMENAME:-${REPOSITORY_NAME,,}_data}
platform=${PLATFORM:-Linux/${arch}} ARCH=${ARCH:-$(arch)}
invokeai_tag=${INVOKEAI_TAG:-${project_name}:${arch}} PLATFORM=${PLATFORM:-Linux/${ARCH}}
gpus=${GPU_FLAGS:+--gpus=${GPU_FLAGS}} CONTAINER_FLAVOR=${CONTAINER_FLAVOR:-cuda}
INVOKEAI_TAG=${REPOSITORY_NAME,,}-${CONTAINER_FLAVOR}:${INVOKEAI_TAG:-latest}
export project_name
export volumename
export arch
export platform
export invokeai_tag
export gpus

28
docker-build/run.sh
View File

@ -1,21 +1,31 @@
#!/usr/bin/env bash #!/usr/bin/env bash
set -e set -e
source ./docker-build/env.sh || echo "please run from repository root" || exit 1 # How to use: https://invoke-ai.github.io/InvokeAI/installation/INSTALL_DOCKER/#run-the-container
# IMPORTANT: You need to have a token on huggingface.co to be able to download the checkpoints!!!
source ./docker-build/env.sh \
|| echo "please run from repository root" \
|| exit 1
# check if HUGGINGFACE_TOKEN is available
# You must have accepted the terms of use for required models
HUGGINGFACE_TOKEN=${HUGGINGFACE_TOKEN:?Please set your token for Huggingface as HUGGINGFACE_TOKEN}
echo -e "You are using these values:\n" echo -e "You are using these values:\n"
echo -e "volumename:\t ${volumename}" echo -e "Volumename:\t ${VOLUMENAME}"
echo -e "invokeai_tag:\t ${invokeai_tag}\n" echo -e "Invokeai_tag:\t ${INVOKEAI_TAG}\n"
docker run \ docker run \
--interactive \ --interactive \
--tty \ --tty \
--rm \ --rm \
--platform="$platform" \ --platform="$PLATFORM" \
--name="$project_name" \ --name="${REPOSITORY_NAME,,}" \
--hostname="$project_name" \ --hostname="${REPOSITORY_NAME,,}" \
--mount="source=$volumename,target=/data" \ --mount="source=$VOLUMENAME,target=/data" \
--env="HUGGINGFACE_TOKEN=${HUGGINGFACE_TOKEN}" \
--publish=9090:9090 \ --publish=9090:9090 \
--cap-add=sys_nice \ --cap-add=sys_nice \
$gpus \ ${GPU_FLAGS:+--gpus=${GPU_FLAGS}} \
"$invokeai_tag" ${1:+$@} "$INVOKEAI_TAG" ${1:+$@}

19
docs/installation/040_INSTALL_DOCKER.md
View File

@ -78,15 +78,16 @@ Some Suggestions of variables you may want to change besides the Token:
<figure markdown> <figure markdown>
| Environment-Variable | Default value | Description | | Environment-Variable | Default value | Description |
| -------------------- | ----------------------------- | -------------------------------------------------------------------------------------------- | | -------------------- | ----------------------------- | -------------------------------------------------------------------------------------------- |
| `HUGGINGFACE_TOKEN` | No default, but **required**! | This is the only **required** variable, without it you can't download the huggingface models | | `HUGGINGFACE_TOKEN` | No default, but **required**! | This is the only **required** variable, without it you can't download the huggingface models |
| `PROJECT_NAME` | `invokeai` | affects the project folder, tag- and volume name | | `REPOSITORY_NAME` | The Basename of the Repo folder | This name will used as the container repository/image name |
| `VOLUMENAME` | `${PROJECT_NAME}_data` | Name of the Docker Volume where model files will be stored | | `VOLUMENAME` | `${REPOSITORY_NAME,,}_data` | Name of the Docker Volume where model files will be stored |
| `ARCH` | `x86_64` | can be changed to f.e. aarch64 if you are using a ARM based CPU | | `ARCH` | arch of the build machine | can be changed if you want to build the image for another arch |
| `INVOKEAI_TAG` | `${PROJECT_NAME}:${ARCH}` | the Container Repository / Tag which will be used | | `INVOKEAI_TAG` | latest | the Container Repository / Tag which will be used |
| `PIP_REQUIREMENTS` | `requirements-lin-cuda.txt` | the requirements file to use (from `environments-and-requirements`) | | `PIP_REQUIREMENTS` | `requirements-lin-cuda.txt` | the requirements file to use (from `environments-and-requirements`) |
| `INVOKE_DOCKERFILE` | `docker-build/Dockerfile` | the Dockerfile which should be built, handy for development | | `CONTAINER_FLAVOR` | cuda | the flavor of the image, which can be changed if you build f.e. with amd requirements file. |
| `INVOKE_DOCKERFILE` | `docker-build/Dockerfile` | the Dockerfile which should be built, handy for development |
</figure> </figure>