diff --git a/docs/index.md b/docs/index.md index 3c5bd3904b..c38f840d32 100644 --- a/docs/index.md +++ b/docs/index.md @@ -93,9 +93,15 @@ getting InvokeAI up and running on your system. For alternative installation and upgrade instructions, please see: [InvokeAI Installation Overview](installation/) -Linux users who wish to make use of the PyPatchMatch inpainting functions will -need to perform a bit of extra work to enable this module. Instructions can be -found at [Installing PyPatchMatch](installation/060_INSTALL_PATCHMATCH.md). +Users who wish to make use of the **PyPatchMatch** inpainting functions +will need to perform a bit of extra work to enable this +module. Instructions can be found at [Installing +PyPatchMatch](installation/060_INSTALL_PATCHMATCH.md). + +If you have an NVIDIA card, you can benefit from the significant +memory savings and performance benefits provided by Facebook Lab's +**xFormers** module. Instructions for Linux and Windows users can be found +at [Installing xFormers](installation/070_INSTALL_XFORMERS.md). ## :fontawesome-solid-computer: Hardware Requirements diff --git a/docs/installation/070_INSTALL_XFORMERS.md b/docs/installation/070_INSTALL_XFORMERS.md new file mode 100644 index 0000000000..be54a3ee86 --- /dev/null +++ b/docs/installation/070_INSTALL_XFORMERS.md @@ -0,0 +1,206 @@ +--- +title: Installing xFormers +--- + +# :material-image-size-select-large: Installing xformers + +xFormers is toolbox that integrates with the pyTorch and CUDA +libraries to provide accelerated performance and reduced memory +consumption for applications using the transformers machine learning +architecture. After installing xFormers, InvokeAI users who have +CUDA GPUs will see a noticeable decrease in GPU memory consumption and +an increase in speed. + +xFormers can be installed into a working InvokeAI installation without +any code changes or other updates. This document explains how to +install xFormers. + +## Pip Install + +For both Windows and Linux, you can install `xformers` in just a +couple of steps from the command line. + +If you are used to launching `invoke.sh` or `invoke.bat` to start +InvokeAI, then run the launcher and select the "developer's console" +to get to the command line. If you run invoke.py directly from the +command line, then just be sure to activate it's virtual environment. + +Then run the following three commands: + +```sh +pip install xformers==0.0.16rc425 +pip install triton +python -m xformers.info output +``` + +The first command installs `xformers`, the second installs the +`triton` training accelerator, and the third prints out the `xformers` +installation status. If all goes well, you'll see a report like the +following: + +```sh +xFormers 0.0.16rc425 +memory_efficient_attention.cutlassF: available +memory_efficient_attention.cutlassB: available +memory_efficient_attention.flshattF: available +memory_efficient_attention.flshattB: available +memory_efficient_attention.smallkF: available +memory_efficient_attention.smallkB: available +memory_efficient_attention.tritonflashattF: available +memory_efficient_attention.tritonflashattB: available +swiglu.fused.p.cpp: available +is_triton_available: True +is_functorch_available: False +pytorch.version: 1.13.1+cu117 +pytorch.cuda: available +gpu.compute_capability: 8.6 +gpu.name: NVIDIA RTX A2000 12GB +build.info: available +build.cuda_version: 1107 +build.python_version: 3.10.9 +build.torch_version: 1.13.1+cu117 +build.env.TORCH_CUDA_ARCH_LIST: 5.0+PTX 6.0 6.1 7.0 7.5 8.0 8.6 +build.env.XFORMERS_BUILD_TYPE: Release +build.env.XFORMERS_ENABLE_DEBUG_ASSERTIONS: None +build.env.NVCC_FLAGS: None +build.env.XFORMERS_PACKAGE_FROM: wheel-v0.0.16rc425 +source.privacy: open source +``` + +## Source Builds + +`xformers` is currently under active development and at some point you +may wish to build it from sourcce to get the latest features and +bugfixes. + +### Source Build on Linux + +Note that xFormers only works with true NVIDIA GPUs and will not work +properly with the ROCm driver for AMD acceleration. + +xFormers is not currently available as a pip binary wheel and must be +installed from source. These instructions were written for a system +running Ubuntu 22.04, but other Linux distributions should be able to +adapt this recipe. + +#### 1. Install CUDA Toolkit 11.7 + +You will need the CUDA developer's toolkit in order to compile and +install xFormers. **Do not try to install Ubuntu's nvidia-cuda-toolkit +package.** It is out of date and will cause conflicts among the NVIDIA +driver and binaries. Instead install the CUDA Toolkit package provided +by NVIDIA itself. Go to [CUDA Toolkit 11.7 +Downloads](https://developer.nvidia.com/cuda-11-7-0-download-archive) +and use the target selection wizard to choose your platform and Linux +distribution. Select an installer type of "runfile (local)" at the +last step. + +This will provide you with a recipe for downloading and running a +install shell script that will install the toolkit and drivers. For +example, the install script recipe for Ubuntu 22.04 running on a +x86_64 system is: + +``` +wget https://developer.download.nvidia.com/compute/cuda/11.7.0/local_installers/cuda_11.7.0_515.43.04_linux.run +sudo sh cuda_11.7.0_515.43.04_linux.run +``` + +Rather than cut-and-paste this example, We recommend that you walk +through the toolkit wizard in order to get the most up to date +installer for your system. + +#### 2. Confirm/Install pyTorch 1.13 with CUDA 11.7 support + +If you are using InvokeAI 2.3 or higher, these will already be +installed. If not, you can check whether you have the needed libraries +using a quick command. Activate the invokeai virtual environment, +either by entering the "developer's console", or manually with a +command similar to `source ~/invokeai/.venv/bin/activate` (depending +on where your `invokeai` directory is. + +Then run the command: + +```sh +python -c 'exec("import torch\nprint(torch.__version__)")' +``` + +If it prints __1.13.1+cu117__ you're good. If not, you can install the +most up to date libraries with this command: + +```sh +pip install --upgrade --force-reinstall torch torchvision +``` + +#### 3. Install the triton module + +This module isn't necessary for xFormers image inference optimization, +but avoids a startup warning. + +```sh +pip install triton +``` + +#### 4. Install source code build prerequisites + +To build xFormers from source, you will need the `build-essentials` +package. If you don't have it installed already, run: + +```sh +sudo apt install build-essential +``` + +#### 5. Build xFormers + +There is no pip wheel package for xFormers at this time (January +2023). Although there is a conda package, InvokeAI no longer +officially supports conda installations and you're on your own if you +wish to try this route. + +Following the recipe provided at the [xFormers GitHub +page](https://github.com/facebookresearch/xformers), and with the +InvokeAI virtual environment active (see step 1) run the following +commands: + +```sh +pip install ninja +export TORCH_CUDA_ARCH_LIST="6.0;6.1;6.2;7.0;7.2;7.5;8.0;8.6" +pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers +``` + +The TORCH_CUDA_ARCH_LIST is a list of GPU architectures to compile +xFormer support for. You can speed up compilation by selecting +the architecture specific for your system. You'll find the list of +GPUs and their architectures at NVIDIA's [GPU Compute +Capability](https://developer.nvidia.com/cuda-gpus) table. + +If the compile and install completes successfully, you can check that +xFormers is installed with this command: + +```sh +python -m xformers.info +``` + +If suiccessful, the top of the listing should indicate "available" for +each of the `memory_efficient_attention` modules, as shown here: + +```sh +memory_efficient_attention.cutlassF: available +memory_efficient_attention.cutlassB: available +memory_efficient_attention.flshattF: available +memory_efficient_attention.flshattB: available +memory_efficient_attention.smallkF: available +memory_efficient_attention.smallkB: available +memory_efficient_attention.tritonflashattF: available +memory_efficient_attention.tritonflashattB: available +[...] +``` + +You can now launch InvokeAI and enjoy the benefits of xFormers. + +### Windows + +To come + + +--- +(c) Copyright 2023 Lincoln Stein and the InvokeAI Development Team