Edward Smith
09/23/2021, 3:44 PM[EcsRunLauncher] Launching run in task arn:aws:ecs:us-east-1:726618033885:task/data-dagster-prod/2bc319b2cb104976836aacf81a6300d8 on cluster arn:aws:ecs:us-east-1:726618033885:cluster/data-dagster-prod
Could not load pipeline definition.
FileNotFoundError: [Errno 2] No such file or directory: 'recommendations/repo.py'
Edward Smith
09/23/2021, 3:48 PMThis launcher will start each run in a new container, using whatever image that you set in theAnd I do haveenvironment variable in your pipeline container (which will usually be the same image as the pipeline container itself)DAGSTER_CURRENT_IMAGE
DAGSTER_CURRENT_IMAGE: "$REGISTRY_URL/deploy_ecs/pipelines"
in my pipelines definition.
My understanding is that since the DAG is displayed in dagit, that the pipelines contains has successfully served $DAGSTER_HOME/recommendations/repo.py
, but, seemingly it isn't found at execution time. Does the run start with $DAGSTER_HOME
has its working directory?Edward Smith
09/23/2021, 4:09 PM\"repository_origin\": {\"__class__\": \"RepositoryPythonOrigin\", \"code_pointer\": {\"__class__\": \"FileCodePointer\", \"fn_name\": \"repository\", \"python_file\": \"recommendations/repo.py\", \"working_directory\": \"/opt/dagster/dagster_home\"},
Which tells me that the working directory is /opt/dagster/dagster_home/
and that it is trying to load recommendations/repo.py
. That should just work, as here it is from within the pipelines
container:
root@63629cdf2b98:/opt/dagster/dagster_home# pwd
/opt/dagster/dagster_home
root@63629cdf2b98:/opt/dagster/dagster_home# ls -al recommendations/repo.py
-rw-r--r-- 1 root root 12523 Sep 23 14:57 recommendations/repo.py
What am I missing here?Edward Smith
09/23/2021, 4:57 PM