Hi everyone... need some understanding of the trouble I'm having with executing my pipeline only by Dagit (with direct python and by dagster all works as expected) Sorry, can't share the code, only screenshots from remote VM... I have the following config, job (graph), and a set of ops, almost all of which use an input resource (some Class with functions from a package for DataLake)... except one - aggregate_files don't use it, it takes 2 strings of local paths, uses pandas to aggregate CSVs in one and outputs the string with a local path for it... the paths like "prp/Bluechip/..." are exactly from DataLake and only paths like "PoC/sandbox/..." are local... the problem with Dagit on a step of aggregating files is also present on a screenshot... and can't figure out from where it occurs... the step is configured only for local manipulations... so, why I'm getting this error about "No such file or directory" with a path from a DataLake (that existent for sure because it also used by previous steps to pull files from DL) ???

Hi @Dimka Filippov. 🙂 You mentioned that aggregate_files is configured only for local manipulations but did you by mistake passed deltalake paths to file1 and file2 and not local as string to the aggregate_files op and using it for local operations ? The path below in error in file Screenshot_20211202_162728.png which is used in aggregate_files op seems like DeltaLake path and not local.

prp/Bluechip/WLB_CFP/2021/11/11/dbo

I can show, few secs please

This is indeed weird. Is the op failing everytime with Dagit UI or it is often but not always ?

with Dagit all attempts were failed

Can you share screenshot for aggregate_files op ?

it is one of the first 6 screenshots

Does it fail on complete re-execution and from failed op also ?

now will try from failed op... previously full re-executions were failed too

Can you try to reload the workspace from Dagit UI and then try executing the Job again ?

yes, will try now

This is really a very primitive way to debug this but can you put

<http://context.log.info|context.log.info>(f"{op name} started")
<http://context.log.info|context.log.info>(f"{op name} ended")

at start and end of the ops ? So that we can check the logs in Dagit Ui whether it has even entered the op to compute ? We need to know where it is even getting that path..

ok, I'll try

Hi. You can try putting the config in dagster.yaml file and check. :-)

thanks for the idea, but it's not working... tried to re-launch Dagit after clearing the UI config field, simple reloading of workspace... config is not filling automatically from file dagster.yaml that is placed in the root of the project... so I'm still searching for this feature

Hi. This might help you.

yes, it's already seen by me, and I'm using a "--config" for dagster CLI execution, but there is no similar option for Dagit to point to the config file... this article also mention that I can use a Dagit's Launchpad to supply a config in YAML format... my requirement is to set up Dagit to automatically pick the config.yaml or dagster.yaml file into the Launchpad, so that I will not need to manually put config into the UI

Another thing you can try is to pass the config to Job (pipeline for legacy) at runtime. Parse the YAML and pass it as a dict. Check out the article for both latest and version 0.12.15 for an example and how-to. Latest - https://docs.dagster.io/_apidocs/jobs 0.12.15 - https://docs.dagster.io/_apidocs/pipeline

oh, nice thank you... now I can bind the already having config in dict format to the job and even no more need to have a separate file with config 😃

Glad you got it working 😄