Our serverless deploys are failing this morning. W...
# dagster-serverless
j
Our serverless deploys are failing this morning. Was there a change, perhaps to the Dockerfile? Is there a version I can pin somewhere to avoid unexpected changes?
Copy code
Docker version 20.10.20, build 9fdeb9c
Step 1/28 : FROM python:3.10-slim
 ---> 28358ad70606
Step 2/28 : COPY *dagster_cloud_pre_install.sh *setup.py *requirements.txt /
 ---> Using cache
 ---> 4158472c0dbf
Step 3/28 : RUN if [ -f "dagster_cloud_pre_install.sh" ]; then         ./dagster_cloud_pre_install.sh;     fi
 ---> Using cache
 ---> b593218f10e5
Step 4/28 : RUN pip install --upgrade pip
 ---> Running in 94769210dae9
Requirement already satisfied: pip in /usr/local/lib/python3.10/site-packages (22.2.2)
Collecting pip
  Downloading pip-22.3-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 63.9 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.2.2
    Uninstalling pip-22.2.2:
ERROR: Could not install packages due to an OSError: [Errno 39] Directory not empty: '/usr/local/lib/python3.10/site-packages/pip-22.2.2.dist-info/'
p
@johann any ideas?
j
No new dockerfile. Looking in to the error
p
Thanks!
j
@James Hale you’re using the default base image? nvm I see python 3.10-slim.
f
Sounds like it's related to pip. Have you tried to comment out the pip upgrade (step 4)?
d
James is this happening for you on multiple builds?
(i.e. does it consistently reproduce)
j
@daniel yes - pinning to dagster-cloud 1.0.13 fixes it, essentially:
Copy code
deploy:tasks:branch:
  image: docker
  stage: deploy
  script:
    - apk add --no-cache --quiet build-base git python3-dev python3 py3-pip
    # Pin dagster-cloud version: <https://dagster.slack.com/archives/C03SMUY0SFQ/p1666361961209939>
    - pip install dagster-cloud==1.0.13 --quiet
    - dagster-cloud serverless deploy ...
j
@James Hale my mistake- 1.0.14 had a change to the Dockerfile that reordered some steps. Our current hypothesis is that
pip install --upgrade pip
has been in the cache, and changing the Dockerfile unearthed some latent issue with it
j
@johann thanks. Considering pip comes installed in the
python:3.8-slim
image (or in my case
python:3.10-slim
) is the upgrade really necessary? I'd personally rather see any python dependencies managed in the upstream image, instead of in the build Dockerfile (e.g., if I want a newer pip, I can build on a newer python image). That way I can pin the upstream image to ensure I know what version of python/pip/py-dev I'm building on.
d
We should check what version is installed by default on 3.8-slim these days - in the past we've seen a lot of problems with older versions of pip messing up pins and dependencies
Perhaps leave the upgrade step as optional, i.e., advise a user to perform the upgrade in
dagster_cloud_pre_install.sh
if they need a newer version. My 2c :)
d
22.0.4 is pretty recent - I think you're probably right that we can just take it out
At one point it was like pip 19 by default and all kinds of havoc ensued, seems like times have changed
d
is
dagster_cloud_pre_install.sh
a way to customize the pre-install step during deployment? I just ran into an issue where I need to run do an apt install prior to pip install for
arcgis
python library.
j
That’s correct- it’s not documented yet, but as you’ve found it we’ll just copy in the script from your build directory (repo root by default) and run it before doing any pip installs etc.
d
awesome I will try that, annoying pip dependencies that won’t just install on linux 🤷
p
(we had docs for this sitting in a PR for a long time, landing them today, sorry!)
d
just tried setting this up, got this error. Trying to add executable
chmod +x
to fix it?
Copy code
> [ 3/11] RUN if [ -f "dagster_cloud_pre_install.sh" ]; then         ./dagster_cloud_pre_install.sh;     fi:
#9 0.122 /bin/sh: 1: ./dagster_cloud_pre_install.sh: Permission denied
j
Docker copies the same file permissions from your repo, you’ll just need it to be executable in the repo and then it should be in the container too
👍 1
d
okay got it working thank you
j
@daniel thank you! https://github.com/dagster-io/dagster-cloud/blob/main/CHANGES.md
• The default Docker image for Serverless no longer runs
pip install --upgrade pip
.
d
That was all @prha but no problem!
dagsir 1