https://dagster.io/ logo
#dagster-cloud
Title
# dagster-cloud
m

Manish Khatri

03/01/2023, 4:07 PM
Hi, we currently use Dagster Cloud and I wanted to know if Dagster supports triggering a job in ECS via the AWS integration (https://docs.dagster.io/_apidocs/libraries/dagster-aws#ecs). The documentation doesn’t really clarify or give much idea on if this is possible, and an example of how? If it does, does it also get log information and show it within Dagster?
d

daniel

03/01/2023, 4:08 PM
Hi Manish - is this serverless or hybrid?
m

Manish Khatri

03/01/2023, 4:08 PM
Hi Daniel, this is serverless. So it’s hosted on dagster owned infra
d

daniel

03/01/2023, 4:09 PM
Are you hoping to trigger a task in ECS in a separate ECS cluster that you control? Or launch a task within a dagster owned ECS cluster?
m

Manish Khatri

03/01/2023, 4:09 PM
The former, we would aim to trigger a job in our own cluster )
d

daniel

03/01/2023, 4:10 PM
I see - I believe that should be possible using the Python boto3 ECS api in an op that you write - but we don't have built-in support for launching arbitrary ECS tasks in Dagster currently
m

Manish Khatri

03/01/2023, 4:11 PM
Ok. So that particular integration is only for hybrid solution that is hosted on it’s own ecs cluster?
d

daniel

03/01/2023, 4:13 PM
Both hybrid and serverless launch an ECS task for each run in an ECS cluster - the main difference between serverless and hybrid is that we host the ECS cluster for you
m

Manish Khatri

03/01/2023, 4:14 PM
Ok thanks for clarifying. Is there any other way we can trigger some sort of containerised application (docker image) from within dagster serverless that uses some sort if integration dagster already provides? Or would this be a custom thing?
d

daniel

03/01/2023, 4:15 PM
Right now that would need to be a custom thing I think, although I could imagine it being a pretty thin layer on top of the boto3 run_task API if you pass in credentials to your own ECS cluster
m

Manish Khatri

03/01/2023, 4:17 PM
Would you know off the top if the effort to getting log outputs would be high?
d

daniel

03/01/2023, 4:18 PM
I think that would probably involve a call out to the cloudwatch API and printing out the results within the op - doesn't sound too difficult via boto
I think if you already have an ECS cluster that's working for you and you're planning on launching ECS tasks there within serverless runs, then ECS hybrid is also worth considering
m

Manish Khatri

03/01/2023, 4:22 PM
We currently don’t, we are looking at workarounds over wanting to run jobs that process our customer data only within Europe, Dagster servers are running in USA only right now.
HI Daniel, side question - is it possible for us to tell dagster to run a Docker image within dagster serverless? The image would be available to pull from our own Docker repository . The question comes from a place where we can separate our own jobs written in Python (or any other language) from the dagster project, but still have it triggered and executed by Dagster.
d

daniel

03/01/2023, 8:55 PM
Currently we need to host any docker images running in serverless in our cluster
But that's 100% possible in hybrid
m

Manish Khatri

03/01/2023, 8:57 PM
Thanks 👍 is there any documentation on how to do this? I couldn’t find it online
d

daniel

03/01/2023, 8:58 PM
documentation on how to run docker images in hybrid?
m

Manish Khatri

03/01/2023, 8:58 PM
Yup
d

daniel

03/01/2023, 8:59 PM
We have ops that let you run arbitrary containers in k8s: https://docs.dagster.io/_apidocs/libraries/dagster-k8s#dagster_k8s.k8s_job_op and docker: https://docs.dagster.io/_apidocs/libraries/dagster-docker#dagster_docker.docker_container_op - we don't yet have an ecs_task_op but it wouldn't be hard to build one on top of the ECS Python API
m

Manish Khatri

03/01/2023, 9:02 PM
Taking a look at
dagster_docker.execute_docker_container
- the parameters include a registry - would this not work on serverless? My hope is that we can put in those details and thus our own images can be pulled and then run in serverless
d

daniel

03/01/2023, 9:03 PM
It's not a feature we currently support in serverless unfortunately - running docker containers on some other server that you are hosting may be an option by customizing the host that it uses to connect to the Docker socket, but we don't support running arbitrary docker containers in our serverless cluster
👍 1
m

Manish Khatri

03/01/2023, 9:04 PM
Ah ok. Thanks for clarifying 🙂
3 Views