Hi Dagster team! I am testing the backfill feature but I am confusing because when I try to launch the backfill from dagit I got this error:

Traceback (most recent call last):

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\poll_compute_logs.py", line 61, in <module>

execute_polling(sys.argv[1:])

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\poll_compute_logs.py", line 53, in execute_polling

with open(ipc_output_file, "w"):

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\s4957336\\AppData\\Local\\Temp\\1\\tmpo_zr47w6\\execute-windows-tail-15bc135d998546caa23116bbff00ceb1'

2021-06-14 02:33:51 - dagster - ERROR - Mexico_CTMS_Corp - a1cb3fdf-19ef-4676-a6b0-b3c7610ec0f9 - 22036 - PIPELINE_FAILURE - Execution of pipeline "Mexico_CTMS_Corp" failed. An exception was thrown during execution.

Exception: Timed out waiting for tail process to start

Stack Trace:

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\api.py", line 762, in pipeline_execution_iterator

for event in pipeline_context.executor.execute(pipeline_context, execution_plan):

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\executor\in_process.py", line 38, in execute

yield from iter(

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\api.py", line 841, in __iter__

yield from self.iterator(

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\plan\execute_plan.py", line 72, in inner_plan_execution_iterator

active_execution.verify_complete(pipeline_context, step.key)

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__

next(self.gen)

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\storage\compute_log_manager.py", line 56, in watch

yield

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__

next(self.gen)

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\storage\local_compute_log_manager.py", line 51, in _watch_logs

yield

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__

next(self.gen)

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\compute_logs.py", line 31, in mirror_stream_to_file

yield pids

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__

next(self.gen)

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\compute_logs.py", line 75, in tail_to_stream

yield pids

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__

next(self.gen)

File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\compute_logs.py", line 104, in execute_windows_tail

raise Exception("Timed out waiting for tail process to start")

I am not sure why dagster is looking into this path

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\s4957336\\AppData\\Local\\Temp\\1\\tmpo_zr47w6\\execute-windows-tail-15bc135d998546caa23116bbff00ceb1'

if I configured the dagster.yaml in this way:

scheduler:
  module: dagster.core.scheduler
  class: DagsterDaemonScheduler
run_launcher:
  module: dagster.core.launcher
  class: DefaultRunLauncher
run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 25
run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_url: "<postgresql://user:pwd@hostname>:port/dagster"

local_artifact_storage:
  module: dagster.core.storage.root
  class: LocalArtifactStorage
  config:
    base_dir: C:\Users\s4957336\Documents\caps\local_artifact_storage

compute_logs:
  module: dagster.core.storage.local_compute_log_manager
  class: LocalComputeLogManager
  config:
    base_dir: C:\Users\username\Documents\caps\logs

The folder logs is storing the compute logs

It could be related to this issue https://github.com/dagster-io/dagster/issues/3898 Specifying the

base_dir

in the dagster.yaml file hasn't been working, at least for me when using Windows OS file system. As a workaround, I have been using the

configured()

api.

Hi @Daniel Kim, thank you. The local_artifact_storage works correct, the issue is with compute_logs but I already fixed. I solved it by configuring the event_log_storage, schedule_storage and run_storage in the same DB.