Hi everyone, I'm trying to work through the local ...
# announcements
j
Hi everyone, I'm trying to work through the local Dask example here: https://docs.dagster.io/latest/deploying/dask/ and I'm getting some errors about using an ephemeral DagsterInstance. When I write the config in the Dagit playground using
filesystem
, everything works great. When I use the Python API and pass the YAML config like
dagster pipeline execute -f dask_hello_world.py -n dask_pipeline -e dask_hello_world.yaml
, I get a
PIPELINE_INIT_FAILURE
error.
More of the error message here:
Copy code
dask_pipeline - dbd1d007-38b9-4d20-a49e-bb196edf0e10 - PIPELINE_INIT_FAILURE - Pipeline failure during initialization of pipeline "dask_pipeline". This may be due to a failure in initializing a resource or logger.
 event_specific_data = {"error": ["dagster.core.errors.DagsterUnmetExecutorRequirementsError: You have attempted to use an executor that uses multiple processes with an ephemeral DagsterInstance. A non-ephemeral instance is needed to coordinate execution between multiple processes.
you just need to set
$DAGSTER_HOME
we should improve that error message
it works in
dagit
because we create a temporary directory based
DagsterInstance
for you if
DAGSTER_HOME
is not set
which certainly makes things confusing
j
Oh there was a mention of
$DAGSTER_HOME
in the error now that you mention it:
Copy code
You can configure your default instance via $DAGSTER_HOME or ensure a valid one is passed when invoking the python APIs.
I just didn't know what it meant because I didn't have to do any of that with
dagit
. Thanks for the explanation!
Worked like a charm 🎉
👍 1