19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hello, I'm looking for info on how to run a task t...
# deployment-ecse
Edward Smith
10/04/2021, 3:45 PMHello, I'm looking for info on how to run a task that requires ~700GB of RAM on AWS. I've gotten everything working on Fargate, but it maxes out at 30GB. My initial attempt to replace the Daemon task with an EC2-equivalent is giving me:
Attempting to access run_id, but it was not provided when constructing the OutputContextd
daniel
10/04/2021, 3:46 PMhi Edward, do you have a full stack trace for the error that you're seeing?
e
Edward Smith
10/04/2021, 3:46 PM Copy code
dagster.core.errors.DagsterInvariantViolationError: Attempting to access run_id, but it was not provided when constructing the OutputContext
File "/usr/local/lib/python3.9/site-packages/dagster/core/execution/plan/execute_plan.py", line 195, in _dagster_event_sequence_for_step
for step_event in check.generator(step_events):
File "/usr/local/lib/python3.9/site-packages/dagster/core/execution/plan/execute_step.py", line 328, in core_dagster_event_sequence_for_step
for evt in _type_check_and_store_output(step_context, user_event, input_lineage):
File "/usr/local/lib/python3.9/site-packages/dagster/core/execution/plan/execute_step.py", line 382, in _type_check_and_store_output
for evt in _store_output(step_context, step_output_handle, output, input_lineage):
File "/usr/local/lib/python3.9/site-packages/dagster/core/execution/plan/execute_step.py", line 492, in _store_output
handle_output_res = output_manager.handle_output(output_context, output.value)
File "/usr/local/lib/python3.9/site-packages/dagster_aws/s3/io_manager.py", line 65, in handle_output
key = self._get_path(context)
File "/usr/local/lib/python3.9/site-packages/dagster_aws/s3/io_manager.py", line 21, in _get_path
return "/".join([self.s3_prefix, "storage", *context.get_output_identifier()])
File "/usr/local/lib/python3.9/site-packages/dagster/core/execution/context/output.py", line 303, in get_output_identifier
run_id = self.run_id
File "/usr/local/lib/python3.9/site-packages/dagster/core/execution/context/output.py", line 137, in run_id
raise DagsterInvariantViolationError(Edward Smith
10/04/2021, 3:53 PMWell, maybe more than one thing is wrong here. The daemon task is failing to start with "botocore.exceptions.NoRegionError: You must specify a region."
Edward Smith
10/04/2021, 6:14 PMIs there any easy path to running tasks larger than 30GB?
y
yuhan
10/04/2021, 7:06 PMHi Edward, how did you execute the run? was it a re-execution, test run via the testing api, or a regular run via api/cli/dagit?
e
Edward Smith
10/05/2021, 4:35 PMHi Yuhan, this was a re-execution, trying to get the step to run on the EC2 host instead of FARGATE. I've really struggled with getting this working and have just decided to run dagster on a large EC2 instance without docker/ECS as it has proven too difficult.
y
yuhan
10/05/2021, 4:37 PMI see. Thanks for the context! Did you manage to run and re-execute stuff with the larger ec2 approach?
e
Edward Smith
10/05/2021, 4:38 PMI haven't been able to get dagster on ECS using EC2 Cluster to work at all.
Edward Smith
10/05/2021, 4:38 PMAfter a bunch of work, I was able to get it working on fargate, but the 30GB limit is way too small for our use case.
y
yuhan
10/05/2021, 5:17 PMit could be there there’s a bug in the io manager or executor you used. do you mind sharing which io managers and executor you used when you hit that error?
e
Edward Smith
10/05/2021, 6:36 PMI have the S3 io manager and the in-process executor configured
y
yuhan
10/05/2021, 8:16 PMgot it. let me try repro it on my end
✅ 1
6 Views