https://dagster.io/ logo
Join Slack
Powered by
Hey all, running into some issues while using a cu...
# ask-community
z
Hey all, running into some issues while using a custom written step launcher. It works when running locally with `dagster dev`however fails when ran in dagster cloud. The biggest problem is that it seems to be occuring outside of user code.
Copy code
dagster._core.errors.DagsterSubprocessError: During multiprocess execution errors occurred in child processes:
In process 263: dagster._core.errors.DagsterUnknownStepStateError: Execution exited with steps {'my_step_launched_asset'} in an unknown state to this process.
This was likely caused by losing communication with the process performing step execution.
Full error in 🧵
Copy code
dagster._core.errors.DagsterSubprocessError: During multiprocess execution errors occurred in child processes:
In process 263: dagster._core.errors.DagsterUnknownStepStateError: Execution exited with steps {'data_axle_poi_raw_from_sftp'} in an unknown state to this process.
This was likely caused by losing communication with the process performing step execution.

Stack Trace:
  File "/venvs/03f91443cf20/lib/python3.9/site-packages/dagster/_core/executor/child_process_executor.py", line 79, in _execute_command_in_child_process
    for step_event in command.execute():
  File "/venvs/03f91443cf20/lib/python3.9/site-packages/dagster/_core/executor/multiprocess.py", line 93, in execute
    yield from execute_plan_iterator(
  File "/venvs/03f91443cf20/lib/python3.9/site-packages/dagster/_core/execution/api.py", line 1101, in __iter__
    yield from self.iterator(
  File "/venvs/03f91443cf20/lib/python3.9/site-packages/dagster/_core/execution/plan/execute_plan.py", line 136, in inner_plan_execution_iterator
    yield from _handle_compute_log_teardown_error(pipeline_context, sys.exc_info())
  File "/venvs/03f91443cf20/lib/python3.9/site-packages/dagster/_core/execution/plan/active.py", line 168, in __exit__
    raise DagsterUnknownStepStateError(

  File "/venvs/03f91443cf20/lib/python3.9/site-packages/dagster/_core/execution/api.py", line 990, in pipeline_execution_iterator
    for event in pipeline_context.executor.execute(pipeline_context, execution_plan):
  File "/venvs/03f91443cf20/lib/python3.9/site-packages/dagster/_core/executor/multiprocess.py", line 306, in execute
    raise DagsterSubprocessError(
The last bit of the error is cropped, and I cant seem to get the any more info in the dagster cloud ui
d
Hey Zach - we might need to take a look at the custom step launcher in order to be able to give a concrete recommendation here, is that something you can share?
z
Spent most of yesterday debugging this and was able to fix it! Seems like the above part was just a bit of a red herring that made it more difficult to figure out the true error. the TLDR is: 1. A change in one of my code locations caused it not to be loaded by my step launcher and raise an import error due to a missing file in the underlying code.zip that’s pushed over. 2. The error wasn’t caught, and so triggered the “finally” block of the 
local_external_step_main.py
, before 
all_events
had any issues. 3. Since the resulting 
events.pkl
had no length, it seemed to trigger an unknown state error. I ended up implementing my own alternative 
local_external_step_main
that catches exceptions from the dagtster provided one, and if an exception does occur, will check for the resulting 
events.pkl
and create a new one with exception details provided if and only if 
events.pkl
represents a null length event list.
79 Views
Open in Slack
PreviousNext
Powered by