https://dagster.io/ logo
#deployment-ecs
Title
# deployment-ecs
r

rectalogic

04/06/2023, 5:40 PM
my agent ECS task is running
dagster/dagster-cloud-agent:1.2.4
and I set
DAGSTER_HOME
env var on the container to
/opt/dagster/dagster_home
. The image for my code location does not have the
/opt/dagster
directory and so I set
DAGSTER_HOME
in
dagster.yaml
env_vars
to
/home/dagster
- but it still seems to be trying to store output in
/opt/dagster
which doesn't exist (and it does not have permission to create)
Copy code
dagster._core.errors.DagsterExecutionHandleOutputError: Error occurred while handling output "result" of step "hello_redshift":
...
The above exception was caused by the following exception:
PermissionError: [Errno 13] Permission denied: '/opt/dagster'
  File "/site/venv/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File "/site/venv/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 439, in iterate_with_context
    next_output = next(iterator)
  File "/site/venv/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 594, in _gen_fn
    gen_output = output_manager.handle_output(output_context, output.value)
  File "/site/venv/lib/python3.10/site-packages/dagster/_core/storage/upath_io_manager.py", line 239, in handle_output
    path.parent.mkdir(parents=True, exist_ok=True)
d

daniel

04/06/2023, 5:43 PM
We can likely fix the place that it's writing to on the filesystem - but are you considering writing these outputs somewhere other than the filesystem at some point by setting up a different io manager (like the s3 io manager)? That will give you some nice things like the ability to re-execute runs from failure, and should fix this as well
But i think you can customize this folder to be someting other than DAGSTER_HOME via this local_artifact_storage key in your dagster.yaml: https://docs.dagster.io/deployment/dagster-instance#local-artifact-storage
r

rectalogic

04/06/2023, 5:51 PM
This is just a "hello world" job that just checks connectivity to our DB. But if it stores outputs in DAGSTER_HOME by default, why is it not using the modified DAGSTER_HOME from my dagster.yaml?
d

daniel

04/06/2023, 5:52 PM
it uses that local_artifact_storage key that I referenced above - I think modifying that will do what you want
DAGSTER_HOME is used to initially load dagster.yaml, it's not used after that
r

rectalogic

04/06/2023, 5:53 PM
oh, and that key was initially set based on the agents DAGSTER_HOME I guess
d

daniel

04/06/2023, 5:53 PM
yeah
r

rectalogic

04/06/2023, 5:55 PM
does the
local_artifact_storage
key go under
user_code_launcher.config
in my dagster.yaml?
d

daniel

04/06/2023, 5:56 PM
local_artifact_storage is a sibling of user_code_launcher
r

rectalogic

04/06/2023, 5:56 PM
thanks
condagster 1
4 Views