Hi In context of the above ^ I m now trying to take the `dbt dagster #deployment-kubernetes

Hi...In context of the above ^ - I'm now trying to...

Jason

01/26/2021, 5:51 PM

Hi...In context of the above ^ - I'm now trying to take the

dbt_example

and fit it into the

k8s-example

as a another User Code Deployment. The documentation on "Deploying on Kubernetes" (for the default architecture) on User Code Deployments mentions that the user-provided image must contain a repo definition and other stuff - make sense. What I'm a bit confused on is 1. What should the actual Dockerfile look like? Should I base it off of the

k8s-example/Dockefile

(plus my required installation)? It has a

# ==> Add user code layer

section 2. Does each user code deployment run its own version of dagster? While dagit and the daemon run on their own version.

johann

01/26/2021, 5:58 PM

Should I base it off of the
k8s-example/Dockefile

Yep, it should have the dagster deps plus your user code

Jason

01/26/2021, 5:59 PM

But I'm also building it so those variables I'll have to setup/hardcode myself depending on my own needs, correct?

johann

01/26/2021, 6:00 PM

Does each user code deployment run its own version of dagster? While dagit and the daemon run on their own version.

By version do you mean release version?

johann

01/26/2021, 6:00 PM

those variables I’ll have to setup/hardcode myself

The docker args?

Jason

01/26/2021, 6:02 PM

Yup, the docker args. I'm assuming my user code is separate from the rest of the deployment, and I need to build it to create the image, then I push it to ECR and have helm pull it in.

Jason

01/26/2021, 6:02 PM

And by "version" i mean the same thing - docker args like

${DAGSTER_VERSION}

Jason

01/26/2021, 6:09 PM

For example, I have a separate folder with my project (use code), plus a Dockerfile. Based on the above I updated it to contain the k8s-example and extra dependencies...

Copy code

ARG BASE_IMAGE
FROM "${BASE_IMAGE}"

ARG DAGSTER_VERSION

# ==> Add Dagster layer
RUN \
# Cron
       apt-get update -yqq \
    && apt-get install -yqq cron \
# Dagster
    && pip install \
        dagster==${DAGSTER_VERSION} \
        dagster-postgres==${DAGSTER_VERSION} \
        dagster-celery[flower,redis,kubernetes]==${DAGSTER_VERSION} \
        dagster-aws==${DAGSTER_VERSION} \
        dagster-k8s==${DAGSTER_VERSION} \
        dagster-celery-k8s==${DAGSTER_VERSION} \
# Cleanup
    &&  rm -rf /var \
    &&  rm -rf /root/.cache  \
    &&  rm -rf /usr/lib/python2.7 \
    &&  rm -rf /usr/lib/x86_64-linux-gnu/guile

# ==> Add user code layer
RUN pip install \
    dagster-pandas \
    dagster-slack \
    dagstermill \
    dagster-dbt \
    dbt

# Example pipelines
COPY build_cache/ /

johann

01/26/2021, 6:10 PM

Yeah- we seek to maintain forwards and backwards compat with user code images, at least on dot releases, so it should be ok if there’s a little bit of drift between the dagster version in you deployment (dagit+daemon) and your user code images

Jason

01/26/2021, 6:19 PM

Great, thank you. I'll test it out

2 Views

Open in Slack

Previous Next