https://dagster.io/ logo
#ask-community
Title
# ask-community
n

Nick Cleveland

05/16/2023, 12:31 AM
I having an issue with how the ecs run launcher assigns values for run task cpu and memory. Currently, I have one fargate container for dagit/daemon, and another container for my pipeline code. I have tried setting cpu/memory values in the dagster.yaml in the same repo as my user code, and I have also tried setting these values as op tags on my asset, but the tasks for my runs will always generate using the same cpu/memory values as the container running dagit/daemon. This is troublesome because I have some pipelines that are memory intensive, and others that are not, so I would rather be able to set these values accordingly. Any suggestions would be appreciated.
d

daniel

05/16/2023, 2:37 AM
Hi Nick - you'll need to apply tags to the job that you're launching for them to be applied, not to your op/asset.
you can see an example of a job tagged with CPU/memory values in ECS here: https://docs.dagster.io/deployment/guides/aws#customizing-cpu-memory-and-ephemeral-storage-in-ecs
n

Nick Cleveland

05/16/2023, 3:57 PM
Is there a way these tags can be applied to an asset materialization job? Ie the ASSET_JOB materializations? These assets would be managed via freshness policy. A related question I have is whether I can specify default run tags for any run launched via the dagit ui. I see that I can add these ecs/cpu and ecs/memory tags via the launchpad in the ui. Is there somewhere I can configure to where these tags are there by default?
plus1 1
r

Rahil Sondhi

07/14/2023, 11:49 PM
i have the same question ^ summary: is there a way to change the cpu/memory of an asset materialization job? use case: • i want to materialize an asset partition (ie one day) • that day has an anomalous amount of rows, and it needs more memory to process • i have to write a git commit to change
container_context.ecs.run_resources.memory
, push, wait for deploy, try again, then revert my commit • i would rather use the dagit UI Launchpad and specify the memory. would be way faster/more convenient
or specifying it in prod deployment settings would be helpful
z

Zach

07/16/2023, 2:16 AM
If I need to provide an override for a manually-ran job I usually specify it in the Launchpad using the "Edit Tags" button at the right of the screen, using the ecs/cpu and ecs/memory tags
r

Rahil Sondhi

07/16/2023, 2:18 AM
hey Zach, slight difference, you're referring to pure jobs, and Nick and I are referring to asset materialization jobs
z

Zach

07/17/2023, 12:17 AM
Still a little confused about the difference - you mean assets that are materialized outside of a group of assets that have been defined with
define_asset_job
?
r

Rahil Sondhi

07/17/2023, 12:18 AM
correct. there's no job. there's just an asset, and in the UI you click the
Materialize
button
z

Zach

07/17/2023, 12:19 AM
I've always kinda wondered what the use case is for not creating a job for those and just using Materialize? My team just makes everything a job, the pure Materialize stuff has always had kinda weird and limited behavior to me