https://dagster.io/ logo
Powered by
#deployment-kubernetes
Title
# deployment-kubernetes
j

Jason

01/26/2021, 5:51 PM
Hi...In context of the above ^ - I'm now trying to take the 
dbt_example
and fit it into the 
k8s-example
as a another User Code Deployment. The documentation on "Deploying on Kubernetes" (for the default architecture) on User Code Deployments mentions that the user-provided image must contain a repo definition and other stuff - make sense. What I'm a bit confused on is 1. What should the actual Dockerfile look like? Should I base it off of the 
k8s-example/Dockefile
(plus my required installation)? It has a 
# ==> Add user code layer
section 2. Does each user code deployment run its own version of dagster? While dagit and the daemon run on their own version.
j

johann

01/26/2021, 5:58 PM
Should I base it off of the 
k8s-example/Dockefile
Yep, it should have the dagster deps plus your user code
j

Jason

01/26/2021, 5:59 PM
But I'm also building it so those variables I'll have to setup/hardcode myself depending on my own needs, correct?
j

johann

01/26/2021, 6:00 PM
Does each user code deployment run its own version of dagster? While dagit and the daemon run on their own version.
By version do you mean release version?
those variables I’ll have to setup/hardcode myself
The docker args?
j

Jason

01/26/2021, 6:02 PM
Yup, the docker args. I'm assuming my user code is separate from the rest of the deployment, and I need to build it to create the image, then I push it to ECR and have helm pull it in.
And by "version" i mean the same thing - docker args like 
${DAGSTER_VERSION}
For example, I have a separate folder with my project (use code), plus a Dockerfile. Based on the above I updated it to contain the k8s-example and extra dependencies...
Copy code
ARG BASE_IMAGE
FROM "${BASE_IMAGE}"

ARG DAGSTER_VERSION

# ==> Add Dagster layer
RUN \
# Cron
       apt-get update -yqq \
    && apt-get install -yqq cron \
# Dagster
    && pip install \
        dagster==${DAGSTER_VERSION} \
        dagster-postgres==${DAGSTER_VERSION} \
        dagster-celery[flower,redis,kubernetes]==${DAGSTER_VERSION} \
        dagster-aws==${DAGSTER_VERSION} \
        dagster-k8s==${DAGSTER_VERSION} \
        dagster-celery-k8s==${DAGSTER_VERSION} \
# Cleanup
    &&  rm -rf /var \
    &&  rm -rf /root/.cache  \
    &&  rm -rf /usr/lib/python2.7 \
    &&  rm -rf /usr/lib/x86_64-linux-gnu/guile

# ==> Add user code layer
RUN pip install \
    dagster-pandas \
    dagster-slack \
    dagstermill \
    dagster-dbt \
    dbt

# Example pipelines
COPY build_cache/ /
j

johann

01/26/2021, 6:10 PM
Yeah- we seek to maintain forwards and backwards compat with user code images, at least on dot releases, so it should be ok if there’s a little bit of drift between the dagster version in you deployment (dagit+daemon) and your user code images
j

Jason

01/26/2021, 6:19 PM
Great, thank you. I'll test it out
Powered by