sephi
06/09/2020, 11:43 AMbash_command_solid
inside a composit_solid
- and need to set a dependancy between the output of the bash_command and another solid in the pipeline
.
We are not able to get an output from the bash_command_solid.
What would be the correct approach for such task?
While researching this issue - I'm trying to understand the bash_script_solid test example -
can you get the return from the bash script?
what does the following line do in the pipeline ?
a()
from dagster_bash import bash_script_solid
from dagster import file_relative_path, pipeline, composite_solid
@composite_solid()
def composite_bash():
a = bash_script_solid(file_relative_path(__file__, 'hello_world.sh'), name='my_script')
return a
@solid()
def print_string(str_print):
print(str_print)
@composite_solid()
def composite_print(str_to_print):
print_string(str_to_print)
@pipeline
def pip():
a = composite_bash()
composite_print(a)
But this is a non starter since there is an issue with the input /output of the composit_solid
composite_solid composite_bash returned problematic value of type <class 'dagster.core.definitions.solid.SolidDefinition'>. Expected return value from invoked solid or dict mapping output name to return values from invoked solids
alex
06/09/2020, 1:30 PMbash_script_solid
is actually a solid factory, so what may fix it for you is just adding ()
to invoke the solid
a = bash_script_solid(file_relative_path(__file__, 'hello_world.sh'), name='my_script')()
sephi
06/09/2020, 1:46 PMbash_script_solid
.
Here is the updated code:
from dagster_bash import bash_script_solid
from dagster import file_relative_path, pipeline, composite_solid, solid
@composite_solid()
def composite_bash()->str:
a = bash_script_solid(file_relative_path(__file__, 'hello_world.sh'), name='my_script')()
return a
@solid()
def print_string(context, str_print):
print(str_print)
@composite_solid()
def composite_print(str_to_print):
print_string(str_to_print)
@pipeline
def pip():
a = composite_bash()
composite_print(a)
Nothing
or is there another option?alex
06/09/2020, 2:16 PMbash_script_solid
returns returns its output, so what you have should workprint
to <http://context.log.info|context.log.info>
if you want to see the entry in the event streamsephi
06/09/2020, 3:04 PMexecute_pipeline
bash_coommand_solid
?
2. should we go down the route that @Tobias Macey did and work directly with execute
?alex
06/10/2020, 1:12 PMlog / materialize some of the processesI dont quite understand what you mean here, but I am guessing yes to use an approach of building the solid you need from the building blocks in the dagster-bash library
sephi
06/11/2020, 5:33 AMHarry Nguyen
10/10/2020, 4:13 AMprint
I'm debugging code in another module which does not have access to contextalex
10/12/2020, 2:46 PM