Next hurdle I've run into running dagster on ECS i...
# deployment-ecs
e
Next hurdle I've run into running dagster on ECS is that my repo is loaded and displayed in Dagit correctly, but when I run the pipeline, I get:
Copy code
[EcsRunLauncher] Launching run in task arn:aws:ecs:us-east-1:726618033885:task/data-dagster-prod/2bc319b2cb104976836aacf81a6300d8 on cluster arn:aws:ecs:us-east-1:726618033885:cluster/data-dagster-prod
Could not load pipeline definition.
FileNotFoundError: [Errno 2] No such file or directory: 'recommendations/repo.py'
I saw in the guide:
This launcher will start each run in a new container, using whatever image that you set in the 
DAGSTER_CURRENT_IMAGE
 environment variable in your pipeline container (which will usually be the same image as the pipeline container itself)
And I do have
DAGSTER_CURRENT_IMAGE: "$REGISTRY_URL/deploy_ecs/pipelines"
in my pipelines definition. My understanding is that since the DAG is displayed in dagit, that the pipelines contains has successfully served
$DAGSTER_HOME/recommendations/repo.py
, but, seemingly it isn't found at execution time. Does the run start with
$DAGSTER_HOME
has its working directory?
I see in the actual task:
Copy code
\"repository_origin\": {\"__class__\": \"RepositoryPythonOrigin\", \"code_pointer\": {\"__class__\": \"FileCodePointer\", \"fn_name\": \"repository\", \"python_file\": \"recommendations/repo.py\", \"working_directory\": \"/opt/dagster/dagster_home\"},
Which tells me that the working directory is
/opt/dagster/dagster_home/
and that it is trying to load
recommendations/repo.py
. That should just work, as here it is from within the
pipelines
container:
Copy code
root@63629cdf2b98:/opt/dagster/dagster_home# pwd
/opt/dagster/dagster_home
root@63629cdf2b98:/opt/dagster/dagster_home# ls -al recommendations/repo.py
-rw-r--r-- 1 root root 12523 Sep 23 14:57 recommendations/repo.py
What am I missing here?
I figured this out... the Run task wasn't using the tagged docker image.
👍 1