https://dagster.io/ logo
#dagster-cloud
Title
# dagster-cloud
s

Sebastian Delgado von Euw

01/30/2023, 10:06 PM
Is it possible to use our own registry to host base images when using serverless? If not, how would you push a new base image to the dagster registry from our deploy workflow in GH?
s

Shalabh Chaturvedi

01/30/2023, 10:19 PM
Hi Sebastian, it is possible to use custom image in serverless. The way to do this is different depending on whether you use fast deploys or not: This is the default if you created a serverless account in the last two months. If you have
ENABLE_FAST_DEPLOYS: 'true'
in your
.github/workflows/deploy.yml
, see Using a different base image... Note that you must install the
dagster-cloud[serverless]
package in the image you upload. If you don't have fast deploys enabled (if you have an older account or disabled fast deploys for some reason), see Disabled PEX-based deploys, under Change the base image.
s

Sebastian Delgado von Euw

01/30/2023, 10:20 PM
that was not my question. I want to know where the base image will be hosted and how to push it to the registry
s

Shalabh Chaturvedi

01/30/2023, 10:28 PM
Sorry I'm not sure I understand correctly. To answer your question directly:
Is it possible to use our own registry to host base images when using serverless?
No, the images run in serverless must be hosted in dagster cloud.
I want to know where the base image will be hosted and how to push it to the registry
Can you tell me a bit more about about what you are trying to do? Our standard quickstart repos have github workflows that build and push your code but without building a new docker image each time. We run it on a standard docker image that we build.
s

Sebastian Delgado von Euw

01/30/2023, 10:33 PM
Sure, our code is hosted in a monorepo withe the following structure:
Copy code
codeA/
codeB/
dagster_folder/
----/dagster_project/
--------/repository.py
--------/assets/
--------/jobs/
----/workspace.yml
----/setup.py
----/dagster_cloud_post_install.sh
----/dbt_project/
dagster_cloud.yml
We would like to use the code located in
codeA
directory (which is in the parent directory of
dagster_project
) in one of our dagster jobs. We currently use the serverless solution and we use the slow deploy option. Whats the best way to do this?
We thought defining a new base image would be the only solution to use custom dependencies likes this but maybe there is an easier solution.
s

Shalabh Chaturvedi

01/30/2023, 10:49 PM
Ah I see. Yes one option is to use a custom base image. You will have to host the base image and specify it using the
base_image:
parameter (screenshot of docs attached). This requires you upload the image to your own registry and allow the github workflow to pull this image. A simpler option might be to just copy the
codeA
directory into the
dagster_project
but let me think about how this can be achieved.
🙏 1
s

Sebastian Delgado von Euw

01/30/2023, 11:02 PM
good idea actually, I think in our deploy script in GH we can copy the
CodeA
folder to
dagster_project
. We just need to add another step before deploying to dagster
s

Shalabh Chaturvedi

01/30/2023, 11:11 PM
2 Views