Akshat Nigam
08/04/2021, 8:27 PMworkspace.yaml
for python_package:
or python_file:
DBT needs it for its SQL Models.
Now, as far as, Dagster is running in in_process_executor
mode, they (Dagster & DBT Solids) can easily access the EFS, as:
1. We have mounted EFS to main Dagster Deployment.
2. All solids will be executed in same Dagster Deployment as a system process, which will also has the access to EFS.
But the question comes, when Dagster will be running in celery_k8s_job_executor
mode, where each solid is executed in a separate ephemeral kubernetes pod.
_*InBecause, the EFS PV/PVC changes are done for main dagster deployment. I am not that much expert in k8s, but is it possible, if those ephemeral kubernetes pod uses the same main helm chart image as base image, which already has a mount with EFS, the DBTs will have access to EFS as well? It will be really great if someone can give some feedback on this. Thanks.mode, will DBT solids which will get executed in a separate ephemeral k8s pods, have access to the EFS?*_celery_k8s_job_executor