sephi
06/21/2020, 7:57 AMbash_command_solid
inside a composite_solid
- and need to set a dependency between the output of a solid
and the bash_command_solid
.
The pseudo code is as follows:
@composite_solid()
def func():
path_to_file = save_file_solid()
res = bash_command_solid(f"Rscript run_process.R {path_to_file}")
return res
The error we receive is as follows:
dagster.core.definitions.events.Failure: Bash command execution failed with output: /tmp/tmpxxxxx line 1: syntax error near unexpected token `newline'\n/tmp/tmpxxxxx: `Rscirpt run_process.R <dagster.core.definitions.composition.InvokedSolidOutputHandle object at 0x7f....>'\n", "label": "intentional-failure", "metadata_entries":[]}
From what we understand the composite_solid
is running within a pipeline and generating a temp path string output from the solid (without running the solid itself).
Running the bash command in a terminal runs flawless.
In other composite_solid
s we are able to create dependencies so I'm guessing that it is related to bash_command_solid
.
What would be the correct approach for such task?schrockn
06/21/2020, 2:02 PMshell_solid
is the closest thing to what you want You may need to make your own slightly different version if you want use save_file_solid unmodified, but it is a lightweight wrapper around the workhorse execute
function in dagster-shell which you are free to use directly