Skip to content

Commit ad7917c

Browse files
ebrebr
authored
Optimized Docker build with support for external working directory (#1544)
* add docker build optimized for size; do not copy models to image useful for cloud deployments. attempts to utilize docker layer caching as effectively as possible. also some quick tools to help with building * add workflow to build cloud img in ci * push cloud image in addition to building * (ci) also tag docker images with git SHA * (docker) rework Makefile for easy cache population and local use * support the new conda-less install; further optimize docker build * (ci) clean up the build-cloud-img action * improve the Makefile for local use * move execution of invoke script from entrypoint to cmd, allows overriding the cmd if needed (e.g. in Runpod * remove unnecessary copyright statements * (docs) add a section on running InvokeAI in the cloud using Docker * (docker) add patchmatch to the cloud image; improve build caching; simplify Makefile * (docker) fix pip requirements path to use binary_installer directory
1 parent 39cca81 commit ad7917c

File tree

5 files changed

+303
-13
lines changed

5 files changed

+303
-13
lines changed

‎.dockerignore

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,26 @@
11
*
22
!backend
3-
!configs
4-
!environments-and-requirements
53
!frontend
6-
!installer
4+
!binary_installer
75
!ldm
86
!main.py
97
!scripts
108
!server
119
!static
1210
!setup.py
11+
!docker-build
12+
!docs
13+
docker-build/Dockerfile
14+
15+
# Guard against pulling in any models that might exist in the directory tree
16+
**/*.pt*
17+
18+
# unignore configs, but only ignore the custom models.yaml, in case it exists
19+
!configs
20+
configs/models.yaml
21+
22+
# unignore environment dirs/files, but ignore the environment.yml file or symlink in case it exists
23+
!environment*
24+
environment.yml
25+
26+
**/__pycache__

‎.github/workflows/build-cloud-img.yml

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
name: Build and push cloud image
2+
on:
3+
workflow_dispatch:
4+
push:
5+
branches:
6+
- main
7+
- development
8+
tags:
9+
- v*
10+
11+
permissions:
12+
contents: read
13+
packages: write
14+
15+
env:
16+
REGISTRY: ghcr.io
17+
IMAGE_NAME: ${{ github.repository }}
18+
19+
jobs:
20+
docker:
21+
strategy:
22+
fail-fast: false
23+
matrix:
24+
# only x86_64 for now. aarch64+cuda isn't really a thing yet
25+
arch:
26+
- x86_64
27+
runs-on: ubuntu-latest
28+
name: ${{ matrix.arch }}
29+
steps:
30+
- name: Checkout
31+
uses: actions/checkout@v3
32+
33+
- name: Docker meta
34+
id: meta
35+
uses: docker/metadata-action@v4
36+
with:
37+
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
38+
tags: |
39+
type=ref,event=branch
40+
type=ref,event=tag
41+
type=ref,event=pr
42+
type=sha
43+
44+
- name: Set up Docker Buildx
45+
uses: docker/setup-buildx-action@v2
46+
47+
- if: github.event_name != 'pull_request'
48+
name: Docker login
49+
uses: docker/login-action@v2
50+
with:
51+
registry: ghcr.io
52+
username: ${{ github.actor }}
53+
password: ${{ secrets.GITHUB_TOKEN }}
54+
55+
- name: Build and push cloud image
56+
uses: docker/build-push-action@v3
57+
with:
58+
context: .
59+
file: docker-build/Dockerfile.cloud
60+
platforms: Linux/${{ matrix.arch }}
61+
push: ${{ github.event_name != 'pull_request' }}
62+
tags: ${{ steps.meta.outputs.tags }}
63+
labels: ${{ steps.meta.outputs.labels }}

‎docker-build/Dockerfile.cloud

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
#######################
2+
#### Builder stage ####
3+
4+
FROM library/ubuntu:22.04 AS builder
5+
6+
ARG DEBIAN_FRONTEND=noninteractive
7+
RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
8+
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
9+
--mount=type=cache,target=/var/lib/apt,sharing=locked \
10+
apt update && apt-get install -y \
11+
git \
12+
libglib2.0-0 \
13+
libgl1-mesa-glx \
14+
python3-venv \
15+
python3-pip \
16+
build-essential \
17+
python3-opencv \
18+
libopencv-dev
19+
20+
# This is needed for patchmatch support
21+
RUN cd /usr/lib/x86_64-linux-gnu/pkgconfig/ &&\
22+
ln -sf opencv4.pc opencv.pc
23+
24+
ARG WORKDIR=/invokeai
25+
WORKDIR ${WORKDIR}
26+
27+
ENV VIRTUAL_ENV=${WORKDIR}/.venv
28+
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
29+
30+
RUN --mount=type=cache,target=/root/.cache/pip \
31+
python3 -m venv ${VIRTUAL_ENV} &&\
32+
pip install --extra-index-url https://download.pytorch.org/whl/cu116 \
33+
torch==1.12.0+cu116 \
34+
torchvision==0.13.0+cu116 &&\
35+
pip install -e git+https://github.com/invoke-ai/PyPatchMatch@0.1.3#egg=pypatchmatch
36+
37+
COPY . .
38+
RUN --mount=type=cache,target=/root/.cache/pip \
39+
cp binary_installer/py3.10-linux-x86_64-cuda-reqs.txt requirements.txt && \
40+
pip install -r requirements.txt &&\
41+
pip install -e .
42+
43+
44+
#######################
45+
#### Runtime stage ####
46+
47+
FROM library/ubuntu:22.04 as runtime
48+
49+
ARG DEBIAN_FRONTEND=noninteractive
50+
ENV PYTHONUNBUFFERED=1
51+
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
52+
--mount=type=cache,target=/var/lib/apt,sharing=locked \
53+
apt update && apt install -y --no-install-recommends \
54+
git \
55+
curl \
56+
ncdu \
57+
iotop \
58+
bzip2 \
59+
libglib2.0-0 \
60+
libgl1-mesa-glx \
61+
python3-venv \
62+
python3-pip \
63+
build-essential \
64+
python3-opencv \
65+
libopencv-dev &&\
66+
apt-get clean && apt-get autoclean
67+
68+
ARG WORKDIR=/invokeai
69+
WORKDIR ${WORKDIR}
70+
71+
ENV INVOKEAI_ROOT=/mnt/invokeai
72+
ENV VIRTUAL_ENV=${WORKDIR}/.venv
73+
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
74+
75+
COPY --from=builder ${WORKDIR} ${WORKDIR}
76+
COPY --from=builder /usr/lib/x86_64-linux-gnu/pkgconfig /usr/lib/x86_64-linux-gnu/pkgconfig
77+
78+
# build patchmatch
79+
RUN python -c "from patchmatch import patch_match"
80+
81+
## workaround for non-existent initfile when runtime directory is mounted; see #1613
82+
RUN touch /root/.invokeai
83+
84+
ENTRYPOINT ["bash"]
85+
86+
CMD ["-c", "python3 scripts/invoke.py --web --host 0.0.0.0"]

‎docker-build/Makefile

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Directory in the container where the INVOKEAI_ROOT (runtime dir) will be mounted
2+
INVOKEAI_ROOT=/mnt/invokeai
3+
# Host directory to contain the runtime dir. Will be mounted at INVOKEAI_ROOT path in the container
4+
HOST_MOUNT_PATH=${HOME}/invokeai
5+
6+
IMAGE=local/invokeai:latest
7+
8+
USER=$(shell id -u)
9+
GROUP=$(shell id -g)
10+
11+
# All downloaded models, config, etc will end up in ${HOST_MOUNT_PATH} on the host.
12+
# This is consistent with the expected non-Docker behaviour.
13+
# Contents can be moved to a persistent storage and used to prime the cache on another host.
14+
15+
build:
16+
DOCKER_BUILDKIT=1 docker build -t local/invokeai:latest -f Dockerfile.cloud ..
17+
18+
configure:
19+
docker run --rm -it --runtime=nvidia --gpus=all \
20+
-v ${HOST_MOUNT_PATH}:${INVOKEAI_ROOT} \
21+
-e INVOKEAI_ROOT=${INVOKEAI_ROOT} \
22+
${IMAGE} -c "python scripts/configure_invokeai.py"
23+
24+
# Run the container with the runtime dir mounted and the web server exposed on port 9090
25+
web:
26+
docker run --rm -it --runtime=nvidia --gpus=all \
27+
-v ${HOST_MOUNT_PATH}:${INVOKEAI_ROOT} \
28+
-e INVOKEAI_ROOT=${INVOKEAI_ROOT} \
29+
-p 9090:9090 \
30+
${IMAGE} -c "python scripts/invoke.py --web --host 0.0.0.0"
31+
32+
# Run the cli with the runtime dir mounted
33+
cli:
34+
docker run --rm -it --runtime=nvidia --gpus=all \
35+
-v ${HOST_MOUNT_PATH}:${INVOKEAI_ROOT} \
36+
-e INVOKEAI_ROOT=${INVOKEAI_ROOT} \
37+
${IMAGE} -c "python scripts/invoke.py"
38+
39+
# Run the container with the runtime dir mounted and open a bash shell
40+
shell:
41+
docker run --rm -it --runtime=nvidia --gpus=all \
42+
-v ${HOST_MOUNT_PATH}:${INVOKEAI_ROOT} ${IMAGE} --
43+
44+
.PHONY: build configure web cli shell

‎docs/installation/INSTALL_DOCKER.md

Lines changed: 93 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ title: Docker
66

77
!!! warning "For end users"
88

9-
We highly recommend to Install InvokeAI locally using [these instructions](index.md)"
9+
We highly recommend to Install InvokeAI locally using [these instructions](index.md)
1010

1111
!!! tip "For developers"
1212

@@ -16,6 +16,10 @@ title: Docker
1616

1717
For general use, install locally to leverage your machine's GPU.
1818

19+
!!! tip "For running on a cloud instance/service"
20+
21+
Check out the [Running InvokeAI in the cloud with Docker](#running-invokeai-in-the-cloud-with-docker) section below
22+
1923
## Why containers?
2024

2125
They provide a flexible, reliable way to build and deploy InvokeAI. You'll also
@@ -36,7 +40,7 @@ development purposes it's fine. Once you're done with development tasks on your
3640
laptop you can build for the target platform and architecture and deploy to
3741
another environment with NVIDIA GPUs on-premises or in the cloud.
3842

39-
## Installation on a Linux container
43+
## Installation in a Linux container (desktop)
4044

4145
### Prerequisites
4246

@@ -117,12 +121,91 @@ also do so.
117121
./docker-build/run.sh "banana sushi" -Ak_lms -S42 -s10
118122
```
119123

120-
This would generate the legendary "banana sushi" with Seed 42, k_lms Sampler and 10 steps.
124+
This would generate the legendary "banana sushi" with Seed 42, k_lms Sampler and 10 steps.
121125

122126
Find out more about available CLI-Parameters at [features/CLI.md](../../features/CLI/#arguments)
123127

124128
---
125129

130+
## Running InvokeAI in the cloud with Docker
131+
132+
We offer an optimized Ubuntu-based image that has been well-tested in cloud deployments. Note: it also works well locally on Linux x86_64 systems with an Nvidia GPU. It *may* also work on Windows under WSL2 and on Intel Mac (not tested).
133+
134+
An advantage of this method is that it does not need any local setup or additional dependencies.
135+
136+
See the `docker-build/Dockerfile.cloud` file to familizarize yourself with the image's content.
137+
138+
### Prerequisites
139+
140+
- a `docker` runtime
141+
- `make` (optional but helps for convenience)
142+
- Huggingface token to download models, or an existing InvokeAI runtime directory from a previous installation
143+
144+
Neither local Python nor any dependencies are required. If you don't have `make` (part of `build-essentials` on Ubuntu), or do not wish to install it, the commands from the `docker-build/Makefile` are readily adaptable to be executed directly.
145+
146+
### Building and running the image locally
147+
148+
1. Clone this repo and `cd docker-build`
149+
1. `make build` - this will build the image. (This does *not* require a GPU-capable system).
150+
1. _(skip this step if you already have a complete InvokeAI runtime directory)_
151+
- `make configure` (This does *not* require a GPU-capable system)
152+
- this will create a local cache of models and configs (a.k.a the _runtime dir_)
153+
- enter your Huggingface token when prompted
154+
1. `make web`
155+
1. Open the `http://localhost:9090` URL in your browser, and enjoy the banana sushi!
156+
157+
To use InvokeAI on the cli, run `make cli`. To open a Bash shell in the container for arbitraty advanced use, `make shell`.
158+
159+
#### Building and running without `make`
160+
161+
(Feel free to adapt paths such as `${HOME}/invokeai` to your liking, and modify the CLI arguments as necessary).
162+
163+
!!! example "Build the image and configure the runtime directory"
164+
```Shell
165+
cd docker-build
166+
167+
DOCKER_BUILDKIT=1 docker build -t local/invokeai:latest -f Dockerfile.cloud ..
168+
169+
docker run --rm -it -v ${HOME}/invokeai:/mnt/invokeai local/invokeai:latest -c "python scripts/configure_invokeai.py"
170+
```
171+
172+
!!! example "Run the web server"
173+
```Shell
174+
docker run --runtime=nvidia --gpus=all --rm -it -v ${HOME}/invokeai:/mnt/invokeai -p9090:9090 local/invokeai:latest
175+
```
176+
177+
Access the Web UI at http://localhost:9090
178+
179+
!!! example "Run the InvokeAI interactive CLI"
180+
```
181+
docker run --runtime=nvidia --gpus=all --rm -it -v ${HOME}/invokeai:/mnt/invokeai local/invokeai:latest -c "python scripts/invoke.py"
182+
```
183+
184+
### Running the image in the cloud
185+
186+
This image works anywhere you can run a container with a mounted Docker volume. You may either build this image on a cloud instance, or build and push it to your Docker registry. To manually run this on a cloud instance (such as AWS EC2, GCP or Azure VM):
187+
188+
1. build this image either in the cloud (you'll need to pull the repo), or locally
189+
1. `docker tag` it as `your-registry/invokeai` and push to your registry (i.e. Dockerhub)
190+
1. `docker pull` it on your cloud instance
191+
1. configure the runtime directory as per above example, using `docker run ... configure_invokeai.py` script
192+
1. use either one of the `docker run` commands above, substituting the image name for your own image.
193+
194+
To run this on Runpod, please refer to the following Runpod template: https://www.runpod.io/console/gpu-secure-cloud?template=vm19ukkycf (you need a Runpod subscription). When launching the template, feel free to set the image to pull your own build.
195+
196+
The template's `README` provides ample detail, but at a high level, the process is as follows:
197+
198+
1. create a pod using this Docker image
199+
1. ensure the pod has an `INVOKEAI_ROOT=<path_to_your_persistent_volume>` environment variable, and that it corresponds to the path to your pod's persistent volume mount
200+
1. Run the pod with `sleep infinity` as the Docker command
201+
1. Use Runpod basic SSH to connect to the pod, and run `python scripts/configure_invokeai.py` script
202+
1. Stop the pod, and change the Docker command to `python scripts/invoke.py --web --host 0.0.0.0`
203+
1. Run the pod again, connect to your pod on HTTP port 9090, and enjoy the banana sushi!
204+
205+
Running on other cloud providers such as Vast.ai will likely work in a similar fashion.
206+
207+
---
208+
126209
!!! warning "Deprecated"
127210

128211
From here on you will find the the previous Docker-Docs, which will still
@@ -135,12 +218,12 @@ also do so.
135218
If you're on a **Linux container** the `invoke` script is **automatically
136219
started** and the output dir set to the Docker volume you created earlier.
137220

138-
If you're **directly on macOS follow these startup instructions**.
221+
If you're **directly on macOS follow these startup instructions**.
139222
With the Conda environment activated (`conda activate ldm`), run the interactive
140223
interface that combines the functionality of the original scripts `txt2img` and
141-
`img2img`:
224+
`img2img`:
142225
Use the more accurate but VRAM-intensive full precision math because
143-
half-precision requires autocast and won't work.
226+
half-precision requires autocast and won't work.
144227
By default the images are saved in `outputs/img-samples/`.
145228

146229
```Shell
@@ -157,8 +240,8 @@ invoke> q
157240
### Text to Image
158241

159242
For quick (but bad) image results test with 5 steps (default 50) and 1 sample
160-
image. This will let you know that everything is set up correctly.
161-
Then increase steps to 100 or more for good (but slower) results.
243+
image. This will let you know that everything is set up correctly.
244+
Then increase steps to 100 or more for good (but slower) results.
162245
The prompt can be in quotes or not.
163246

164247
```Shell
@@ -172,8 +255,8 @@ You'll need to experiment to see if face restoration is making it better or
172255
worse for your specific prompt.
173256

174257
If you're on a container the output is set to the Docker volume. You can copy it
175-
wherever you want.
176-
You can download it from the Docker Desktop app, Volumes, my-vol, data.
258+
wherever you want.
259+
You can download it from the Docker Desktop app, Volumes, my-vol, data.
177260
Or you can copy it from your Mac terminal. Keep in mind `docker cp` can't expand
178261
`*.png` so you'll need to specify the image file name.
179262

0 commit comments

Comments
 (0)