hi! i have a probably basic question that's specific to deploying on ECS: we just successfully spun up Dagster for the first time, and i see that assets need to be created as python files. is there an easy way to create and edit these files from Dagster's UI? or is the main way people iterate on their setup basically making edits in the S3 bucket?

Hello, I successfully setup Dagster in AWS using https://github.com/dagster-io/dagster/tree/1.2.1/examples/deploy_ecs . But now I realise this is a bit of a dated example as it still uses

repo.py

. Then I had a look at this repo: https://github.com/dagster-io/dagster/tree/master/examples/quickstart_aws . Which seems to be more up to date but it does not have any docs on how to deploy it on aws. Is there an up to date example on how to deploy the open source to AWS with the

Definitions

way of working? 🙂

hey all – we've trying to deploy dagster on ECS, but we've struggled to find a recent enough template to go off of. the guide from the dagster repo (link) says "_Once you're done experimenting with this reference deployment, you should destroy the AWS resources it creates._", so i can't whether this is meant to be used in production. are there any good recent deployment templates that you'd recommend following?

📗 resolved Hey All, we are running Dagster Cloud hybrid setup, when i added sample dagster example from UI using below config

location_name: cloud-examples
image: dagster/dagster-cloud-examples
code_source:
  package_name: dagster_cloud_examples

I launched the job and it run fine and provided me the results, however the task is till there and consuming resources on AWS. I understand that Agent needs to run all the time however, code location specific ecs service/task should only be triggered when we push run or scheduler runs it. is the functionality or am i missing something ?

Hello, we have deployed Dagster on ECS and currently the setup reads passwords from secrets manager to access the db. From what I gather by reading the docs, it seems that any job will thus have access to the same secrets. As I am setting up dagster to be used by many projects, it would be good if we can configure the permissions in finer grain. For example, the data engineers might be building pipeline to redact sensitive information from the source. While the ML engineers might be able to use the redacted information, but blocked from reading the raw data with sensitive info. My concern is since the same secrets can be accessed by any jobs, the raw info accessed by the data engineer will hence be accessible by the ML engineer, which then defeat the purpose of redaction. Is there an easy way to configure this? Or is it currently not possible?

I'm using Service Discovery/CloudMap in ECS to make a code location service discoverable, but it's with a network mode that requires a SRV DNS record instead of A. Getting a DNS resolution failed qtype=A failure from Dagster's GRPC. Assume it's not looking for the right DNS record. Does this strike anyone deploying to ECS as familiar?

Is there any documentation on using AWS ECR to store container images for Dagster jobs? or any documentation on what is required for Dagster to communicate with and use AWS ECR?

if I use a location that references the

latest

tag of an image - what will trigger dagster to force a new version of the

TaskDefinition

so it pulls the new image?

Hopefully last time I have to post here 🙂 . We are trying to use the ECS Run Launcher to run our jobs in ECS however I get the following error when launching the run in launchpad...

TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_graphql/implementation/utils.py", line 126, in _fn
    return fn(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_graphql/implementation/execution/launch_execution.py", line 35, in launch_pipeline_execution
    return _launch_pipeline_execution(graphene_info, execution_params)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_graphql/implementation/execution/launch_execution.py", line 69, in _launch_pipeline_execution
    run = do_launch(graphene_info, execution_params, is_reexecuted)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_graphql/implementation/execution/launch_execution.py", line 54, in do_launch
    return graphene_info.context.instance.submit_run(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/instance/__init__.py", line 2079, in submit_run
    submitted_run = self._run_coordinator.submit_run(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/run_coordinator/default_run_coordinator.py", line 40, in submit_run
    self._instance.launch_run(pipeline_run.run_id, context.workspace)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/instance/__init__.py", line 2132, in launch_run
    self.run_launcher.launch_run(LaunchRunContext(dagster_run=run, workspace=workspace))
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_aws/ecs/launcher.py", line 370, in launch_run
    run_task_kwargs = self._run_task_kwargs(run, image, container_context)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_aws/ecs/launcher.py", line 557, in _run_task_kwargs
    self._get_current_task(),
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_aws/ecs/launcher.py", line 483, in _get_current_task
    current_task_metadata = self._get_current_task_metadata()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_aws/ecs/launcher.py", line 478, in _get_current_task_metadata
    self._current_task_metadata = get_current_ecs_task_metadata()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_aws/ecs/tasks.py", line 255, in get_current_ecs_task_metadata
    task_metadata_uri = _container_metadata_uri() + "/task"

Our config block in the dagster.yaml (AWS acct redacted) is...

run_launcher:
  module: "dagster_aws.ecs"
  class: "EcsRunLauncher"
  config:
    use_current_ecs_task_config: false
    run_task_kwargs:
      cluster: arn:aws:ecs:us-west-2:XXXXXXXXX:cluster/produsa-data-science
      launchType: "FARGATE"

Am I missing something in the config? I can post the code for the actual job if needed but figured I would start with this first.

I have egress locked down in my ECS cluster. When I try to add a code location, the new task it spawns eventually fails with this error. This seems like a network issue not permissions, it launches the new task 5 times and each time it fails then it gives up. Is their some GRPC port or something that I need to allow egress on between the agent task and the code location task?

dagster_cloud.workspace.ecs.client.EcsServiceError: ECS service failed because task arn:aws:ecs:us-west-2:XXXX:task/dagster-cluster/9cd06ec3d77940d4a3032497b3b58561 failed: ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried 5 time(s): failed to fetch secret arn:aws:secretsmanager:us-west-2:XXXX:secret:XXXXDatascienceSecrets from secrets manager: RequestCanceled: request context canceled caused by: context deadline exceeded. Please check your task network configuration.

  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1255, in _reconcile
    self._wait_for_new_server_ready(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 359, in _wait_for_new_server_ready
    task_arn = self.client.wait_for_new_service(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 405, in wait_for_new_service
    return self.check_service_has_running_task(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 517, in check_service_has_running_task
    self._raise_failed_task(task, container_name, logger)
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 442, in _raise_failed_task
    raise EcsServiceError(task_arn=task_arn, stopped_reason=stopped_reason, logs=logs)

my agent ECS task is running

dagster/dagster-cloud-agent:1.2.4

and I set

DAGSTER_HOME

env var on the container to

/opt/dagster/dagster_home

. The image for my code location does not have the

/opt/dagster

directory and so I set

DAGSTER_HOME

in

dagster.yaml

env_vars

to

/home/dagster

- but it still seems to be trying to store output in

/opt/dagster

which doesn't exist (and it does not have permission to create)

dagster._core.errors.DagsterExecutionHandleOutputError: Error occurred while handling output "result" of step "hello_redshift":
...
The above exception was caused by the following exception:
PermissionError: [Errno 13] Permission denied: '/opt/dagster'
  File "/site/venv/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File "/site/venv/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 439, in iterate_with_context
    next_output = next(iterator)
  File "/site/venv/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 594, in _gen_fn
    gen_output = output_manager.handle_output(output_context, output.value)
  File "/site/venv/lib/python3.10/site-packages/dagster/_core/storage/upath_io_manager.py", line 239, in handle_output
    path.parent.mkdir(parents=True, exist_ok=True)

Hello all! We are having issues with the Daemon not being able to handle the large number of runs it has to enqueue, whereas the Daemon does not seem to have any issues with dequeuing. We have set the Daemon (Fargate) task CPU to 8192 and the Memory to 16384 (which according to CloudWatch is quite an overkill already). Furthermore, we have enabled threads in both the enqueuer (schedule) and the dequeuer, which are both set to 16. Does anyone else has/had similar issues? Any suggestions on how to improve the Daemon performance? Some more information: • We currently got about 10 active schedules • Schedule evaluations are triggered every 5 minutes • Per schedule evaluation we yield approximately 50-300 Dagster jobs (depending on the schedule) • We see a lot of schedules being stuck on "started" and yielding Dagster runs rather slowly (1-5 per minute)

I'm trying to scope down some of the IAM permissions the dagster cloud ECS agent needs. It wants permission

servicediscovery:DeleteService

to delete any servicediscovery service. Is it possible to use a tag condition key with this

aws:ResourceTag/${TagKey}

? It looks like dagster cloud may pass some tags like

dagster/deployment_name

when it creates the service registry, but not when it deletes it so I suspect there is no way to scope this down. https://docs.aws.amazon.com/service-authorization/latest/reference/list_awscloudmap.html#awscloudmap-service

Hi. I’m new to Dagster and have been trying to reconfigure Dagster containers in ECS but I’m running into problems. For our team Dagster is currently running on EC2. I have not reconfigured any of the dagster code deployed on EC2 yet for ECS, and perhaps this is causing of some of the problems. For the most part, I’ve been following the multi-container code examples in the dagster/examples/deploy_ecs. I’ve configured 3 services/containers: dagit, daemon and user_code. Each service has its own task definition. The services and task definitions are defined using terraform. I’ve configured ECS Service Connect to connect the 3 services in ECS. I’m able to get the user_code container to deploy and run successfully but the daemon (dagster-daemon) and dagit containers stop running fairly soon after start-up with an exit code of 0. One of my questions is, shouldn’t all the containers be running? I have the target group for my ALB configured with a health check for the dagit container on port 3000 for grpc, which results in a healthy state for the ecs service connect endpoint for the user_code container. My goal is get the configuration up and running so that one can connect via the Route 53 A record to the dagit UI. At present, there is a successful connection (i’ve double checked routes and security groups- all good). however, I’m getting a blank page after connecting. Could someone please direct me to what I’m doing wrong, or perhaps point me to documentation that would help to fix the issues I’m having. Will be glad to share dagster.yaml, docker-compose.yml, Dockerfile, etc.

We pull-thru cache the

dagster/dagster-cloud-agent:1.2.4

agent in our private ECR. Amazon Inspector is flagging a bunch of vulnerabilities. How often does dagster rebuild tagged images on top of the latest base image to incorporate security patches?

Hi, I set up the example ECS deployment yesterday which went smoothly. As Dagit is available on port 3000 of the Load Balancer's URL which is public - What is the best way of setting it up so only private access is available?

This message was deleted.

Hi! Was wondering, what are some of your thoughts regarding running Dagster against ECS vs EKS? We're testing out right now and would love to hear other people's experiences.

Hi! I'm looking to migrate the dagster instance to ECS, I saw that there is documentation for Kubernetes but not for ECS. What would be the procedure to follow?

We had a dagster deployment working well on ECS, with a container for dagit and one for the daemon. Then, we upgraded the dagster version and can't get the service to stay healthy in ECS no matter what. Were there any recent breaking changes related to ECS? Are there canonical examples of the best options to use for health check commands? The weird thing is that dagit runs fine, briefly, and then for some reason ECS kills it. Would love an example of people's health check cmds - I'm just changing it to

dagster -v

while debugging to figure out what's up

I get code server stats and stops for no clear reason..

5/10/2023, 5:43:22 PM GMT+1	[32m2023-05-10 16:43:22 +0000[0m - dagster - [34mINFO[0m - Shutting down Dagster code server for file /opt/dagster/dagster_home/modules/.../module.py in process 5807	dbd67f391a7a47858ad4f0459c3ff6ba	dagit
5/10/2023, 5:43:22 PM GMT+1	[32m2023-05-10 16:43:22 +0000[0m - dagster - [34mINFO[0m - Shutting down Dagster code server for file /opt/dagster/dagster_home/modules/.../module.py in process 3479	dbd67f391a7a47858ad4f0459c3ff6ba	dagit
5/10/2023, 5:43:22 PM GMT+1	[32m2023-05-10 16:43:22 +0000[0m - dagster - [34mINFO[0m - Shutting down Dagster code server for file /opt/dagster/dagster_home/modules/.../module.py in process 1093	dbd67f391a7a47858ad4f0459c3ff6ba	dagit
5/10/2023, 5:43:22 PM GMT+1	[32m2023-05-10 16:43:22 +0000[0m - dagster - [34mINFO[0m - Shutting down Dagster code server for file /opt/dagster/dagster_home/modules/.../module.py in process 3356	dbd67f391a7a47858ad4f0459c3ff6ba	dagit
5/10/2023, 5:41:47 PM GMT+1	[32m2023-05-10 16:41:47 +0000[0m - dagit - [34mINFO[0m - Serving dagit on <http://0.0.0.0:8080> in process 1	5bea78746a3f44da86fbbbbee75b84ca	dagit
5/10/2023, 5:41:47 PM GMT+1	Telemetry:	5bea78746a3f44da86fbbbbee75b84ca	dagit
5/10/2023, 5:41:47 PM GMT+1	As an open source project, we collect usage statistics to inform development priorities. For more	5bea78746a3f44da86fbbbbee75b84ca	dagit
5/10/2023, 5:41:47 PM GMT+1	information, read <https://docs.dagster.io/install#telemetry>.

But it seems like that is common from searching Slack

Hi All, We are facing a lot of GraphQL errors in Dagit due to the large number of underlying data stored in the PostgreSQL database (see screenshot for the details). I have already increased the statement timeout for Dagit, however, this does not solve the underlying problem! Is there a way to set data retention for the "runs" table created by Dagster inside the PostgreSQL database or some other solution in order to avoid the GraphQL errors to happen (implementing a more specific LIMIT inside the query for instance, not sure what the current LIMIT does as there is no information regarding "param_1")? *Note that this GraphQL error not only occur at the schedules, but almost everywhere within the Dagit UI! Would love to hear your reaction to this! Kind regards, Arnoud

REPOST OF OLD THREAD: Hi @jordan, We have created a procedure that Dagster automatically uses the on-demand instances when no spot instances are available (working towards a PR). However, we were wondering if we can further improve this by requeuing the Dagster runs when no spot instances were available (useful for Dagster runs that have no rush for instance). Hence the following question: Is there a (relatively) simple way to re-enqueue Dagster runs that were rejected due to lack of available spot instances so we can handle this inside the dagster-ecs launch_run definition? We are willing to develop this part, however, some pointers in the right direction would help us a lot in order to speed up the development process! Thank you in advance, Arnoud

Hello folks I am working on upgrading from 1.1.9/0.17.9 to 1.3.4/0.19.4. While iterating through the versions released since 1.1.9/0.17.9 I found that 1.2.3/0.18.3 breaks my schedule and sensor job runs while leaving job runs through the launchpad in tact. I will share more details in the thread.

Hi 👋🏽 . We are using the hybrid cloud. I am wondering if there is anything similar to airflow’s ECS operators? This has been an effective solution for us in the past where teams are in charge of creating/updating ECR images and the job orchestrators (ie airflow) running the images using ECS (or aws batch). I see that e.g pyspark_emr_step_launcher has similar vocabulary (launch_step, wait etc). Hopefully there is a readymade solution already 🤞🏽

Hello! I am attempting to get Dagster setup in ECS using docker compose. I am having an issue where if I setup secrets to be retrieved from AWS Secret Manager in the dagster.yaml file and then calling those as EnvVar for Resources, in my Definitions, it does not launch an instance for the job. It believes the environment variables are missing, but they get created only on the launch instance that is meant to run the job. Is there some sort of work around for this?

Hi all - we recently saw that “Docker Compose’s integration for ECS and ACI is retiring in November 2023.” Without docker compose, what would be the best practice for deploying on ECS? Are there any other examples we can check out? 🙂

Trying dagster for the first time, and deploying to ECS. The example on the dagster github worked perfectly. However, when I try to restrict access to only specific IPs, it doesn’t work. I recognize this is more likely a general-aws issue rather than a dagster issue, but wondering if anyone else had this issue. I went to the network load balancer’s security group sg-<> - dagster-DefaultNetwork-<> And removed the allow anyone rule, and added my own IP (as well as IP of colleagues) in both IPV4 and IPV6, and can’t access Dagster. When I re-add the 0.0.0.0 rule, I can access dagster again

I'd like to understand dagster ECSrunlauncher and what it means by "task" - if I materialize an asset that depends on upstream assets A->B->C - if I have a singleton created in A - would C be able to access the state generated in A? Or is each asset materialization run on a different ECS task - and C can't know what A created in memory. How do we securely keep auth tokens generated in A's task for C? We can't auth too many times (the api we are working with will not permit it). Even if we made a resource rather than a singleton - the state wouldn't persist across ecs tasks. How are people solving this issue? Posted in #dagster-support