https://dagster.io/ logo
Powered by
Title
c

chrispc

07/21/2021, 4:55 PM
Hi Dagster team! I am getting this error several times. I am trying to launch a pipeline from dagit I got this error: 
Traceback (most recent call last):
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\poll_compute_logs.py", line 61, in <module>
execute_polling(sys.argv[1:])
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\poll_compute_logs.py", line 53, in execute_polling
with open(ipc_output_file, "w"):
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\s4957336\\AppData\\Local\\Temp\\1\\tmpo_zr47w6\\execute-windows-tail-15bc135d998546caa23116bbff00ceb1'
2021-06-14 02:33:51 - dagster - ERROR - Mexico_CTMS_Corp - a1cb3fdf-19ef-4676-a6b0-b3c7610ec0f9 - 22036 - PIPELINE_FAILURE - Execution of pipeline "Mexico_CTMS_Corp" failed. An exception was thrown during execution.
Exception: Timed out waiting for tail process to start
Stack Trace:
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\api.py", line 762, in pipeline_execution_iterator
for event in pipeline_context.executor.execute(pipeline_context, execution_plan):
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\executor\in_process.py", line 38, in execute
yield from iter(
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\api.py", line 841, in __iter__
yield from self.iterator(
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\plan\execute_plan.py", line 72, in inner_plan_execution_iterator
active_execution.verify_complete(pipeline_context, step.key)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__
next(self.gen)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\storage\compute_log_manager.py", line 56, in watch
yield
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__
next(self.gen)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\storage\local_compute_log_manager.py", line 51, in _watch_logs
yield
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__
next(self.gen)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\compute_logs.py", line 31, in mirror_stream_to_file
yield pids
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__
next(self.gen)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\compute_logs.py", line 75, in tail_to_stream
yield pids
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__
next(self.gen)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\compute_logs.py", line 104, in execute_windows_tail
raise Exception("Timed out waiting for tail process to start")
I am not sure why dagster is looking into this path 
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\s4957336\\AppData\\Local\\Temp\\1\\tmpo_zr47w6\\execute-windows-tail-15bc135d998546caa23116bbff00ceb1'
 what could I do? I am using windows 10, 64 bit. Conda enviroment. Dagster 0.12.2 version. I got a timeout several times randomnly. I configured the backend in a postgres Instance in dagster.yml file.
a

alex

07/21/2021, 5:03 PM
cc @prha
c

chrispc

07/21/2021, 5:07 PM
Thanks @alex I am trying to convince the architect to migrate our workflows to Dagster but I can't find the solution to this issues in windows. In linux server this error doesn't appear frequently but Windows is more often.
p

prha

07/21/2021, 5:12 PM
This looks like an error from capturing the compute logs from your pipeline execution. I think we can do a better job of not raising an Exception here to ensure that log capture failures don’t kill pipeline execution.
I think I can get something in for the next release (tomorrow), but in the meantime, you can try to switch the instance configuration in your 
dagster.yaml
to use the 
NoopComputeLogManager
. This will raw cause compute logs (stdout/stderr from your pipeline runs) to get discarded, but will ensure that this particular error won’t interrupt your pipeline execution. The corresponding 
dagster.yaml
config entry for that looks like this: 
compute_logs:
  module: dagster.core.storage.noop_compute_log_manager
  class: NoOpComputeLogManager
@Dagster Bot issue Compute logs intermittently raise exception during tail_polling on Windows
d

Dagster Bot

07/21/2021, 5:17 PM
Created issue at: https://github.com/dagster-io/dagster/issues/4383
c

chrispc

07/21/2021, 5:24 PM
@alex @prha Thank you so much. Also, I would like to know how PYTHONLEGACYWINDOWSSTDIO env variable should be declared. I was looking in Slack and Dagster docs but I don't find any related info. @dansasbu fyi!
@prha I apply the changes you said but I got another error related with DagsterIPCProtocol PIPELINE_FAILURE Caught an error for run f56a9e65-99bf-41e5-a2da-725e4362ae2d while removing it from the queue. Marking the run as failed and dropping it from the queue: dagster.serdes.ipc.DagsterIPCProtocolError: Timeout: read stream has not received any data in 60 seconds 
Stack Trace:
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\host_representation\grpc_server_registry.py", line 177, in _get_grpc_endpoint
server_process = GrpcServerProcess(
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\grpc\server.py", line 1065, in __init__
self.server_process, self.port = open_server_process_on_dynamic_port(
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\grpc\server.py", line 1013, in open_server_process_on_dynamic_port
server_process = open_server_process(
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\grpc\server.py", line 990, in open_server_process
wait_for_grpc_server(server_process, output_file)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\grpc\server.py", line 922, in wait_for_grpc_server
event = read_unary_response(ipc_output_file, timeout=timeout, ipc_process=server_process)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\serdes\ipc.py", line 39, in read_unary_response
messages = list(ipc_read_event_stream(output_file, timeout=timeout, ipc_process=ipc_process))
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\serdes\ipc.py", line 152, in ipc_read_event_stream
raise DagsterIPCProtocolError(
the dagster-deamon generates this:
p

prha

07/21/2021, 7:46 PM
@chrispc Is there some reason why we wouldn’t be able to write to that temp directory (either disk full or some permissions error?)
c

chrispc

07/21/2021, 7:54 PM
I am working in the computer of the company and It has some restrictions but I can create file there. Actually I was looking in the dir path showed in the error, I found that folder doesn't exits. It was working yesterday, I just updated Dagster to 0.12.2 version, after that this issue is generated. This is the structure of my temp folder on my disk @prha I can launch the pipeline using the API, the error happens when I try to use dagit and daemon.
7 Views
#dagster-supportJoin Slack
Powered by