Paul Murray
12/08/2022, 3:15 PM@op
def op_one():
return foo
@op
def op_two(foo):
return foo + bar
@job(tags={
"ecs/cpu": "1024",
"ecs/memory": "6144",
},)
def ops_job():
return op_two(op_one())
This worked fine when triggered from Launchpad or scheduled.
I tried setting up the same process as an asset like:
@asset
def my_asset(group_name="my_assets"):
# same logic as the ops
return foo + bar
define_asset_job(
name="asset_job",
selection=AssetSelection.groups("my_assets"),
tags={"ecs/cpu": "1024", "ecs/memory": "6144"},
)
I triggered a run using the materialize asset button. The asset job failed at the same point my ops job did before I increased the memory. My assumption is that the launcher is not registering my custom cpu and memory settings for the asset materialization.
I encountered a case previously where the materialize asset button just didn't work, but other run triggers did, so I'm wondering if that's also what's happening here or is there a proper way to set up the ECS tags for an asset job? Thanks.daniel
12/08/2022, 4:12 PMPaul Murray
12/08/2022, 5:26 PMdaniel
12/08/2022, 5:26 PMPaul Murray
12/08/2022, 5:28 PMdaniel
12/08/2022, 5:37 PMPaul Murray
12/08/2022, 5:40 PM