Is it possible to use our own registry to host base images when using serverless? If not, how would you push a new base image to the dagster registry from our deploy workflow in GH?

Hi Sebastian, it is possible to use custom image in serverless. The way to do this is different depending on whether you use fast deploys or not: This is the default if you created a serverless account in the last two months. If you have

ENABLE_FAST_DEPLOYS: 'true'

in your

.github/workflows/deploy.yml

, see Using a different base image... Note that you must install the

dagster-cloud[serverless]

package in the image you upload. If you don't have fast deploys enabled (if you have an older account or disabled fast deploys for some reason), see Disabled PEX-based deploys, under Change the base image.

that was not my question. I want to know where the base image will be hosted and how to push it to the registry

Sorry I'm not sure I understand correctly. To answer your question directly:

Is it possible to use our own registry to host base images when using serverless?

No, the images run in serverless must be hosted in dagster cloud.

I want to know where the base image will be hosted and how to push it to the registry

Can you tell me a bit more about about what you are trying to do? Our standard quickstart repos have github workflows that build and push your code but without building a new docker image each time. We run it on a standard docker image that we build.

Sure, our code is hosted in a monorepo withe the following structure:

codeA/
codeB/
dagster_folder/
----/dagster_project/
--------/repository.py
--------/assets/
--------/jobs/
----/workspace.yml
----/setup.py
----/dagster_cloud_post_install.sh
----/dbt_project/
dagster_cloud.yml

We would like to use the code located in

codeA

directory (which is in the parent directory of

dagster_project

) in one of our dagster jobs. We currently use the serverless solution and we use the slow deploy option. Whats the best way to do this?

Ah I see. Yes one option is to use a custom base image. You will have to host the base image and specify it using the

base_image:

parameter (screenshot of docs attached). This requires you upload the image to your own registry and allow the github workflow to pull this image. A simpler option might be to just copy the

codeA

directory into the

dagster_project

but let me think about how this can be achieved.

good idea actually, I think in our deploy script in GH we can copy the

CodeA

folder to

dagster_project

. We just need to add another step before deploying to dagster

Yes possibly here, after the checkout: https://github.com/dagster-io/quickstart-etl/blob/main/.github/workflows/deploy.yml#L58-L61