I think there may be some issues with the dagster-...
# announcements
o
I think there may be some issues with the dagster-bash solids. I am on 0.7.5. When I run a basic pipeline, using the bash_script_solid, I am seeing the solid executed on when loading the repository into dagit, as startup. My solid definition looks something like the following:
Copy code
@pipleine
def my_pipeline():
    run_cmd_in_venv = bash_script_solid(
        '.../dagster_pipelines/hello_world.sh',
        name = 'hello'
    )

    execute_solid(
        run_cmd_in_venv
    )
the output on starting dagit is as follows:
Copy code
Loading repository...
2020-04-03 13:29:16 - dagster - DEBUG - ephemeral_hello_solid_pipeline - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - PIPELINE_START - Started execution of pipeline "ephemeral_hello_solid_pipeline".
2020-04-03 13:29:16 - dagster - DEBUG - ephemeral_hello_solid_pipeline - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - ENGINE_EVENT - Executing steps in process (pid: 29701)
 event_specific_data = {"error": null, "marker_end": null, "marker_start": null, "metadata_entries": [["pid", null, ["29701"]], ["step_keys", null, ["['hello.compute']"]]]}
            step_key = "hello.compute"
2020-04-03 13:29:16 - dagster - DEBUG - ephemeral_hello_solid_pipeline - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - STEP_START - Started execution of step "hello.compute".
               solid = "hello"
    solid_definition = "hello"
            step_key = "hello.compute"
2020-04-03 13:29:16 - dagster - INFO - system - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - using temporary directory: /tmp
               solid = "hello"
    solid_definition = "hello"
            step_key = "hello.compute"
2020-04-03 13:29:16 - dagster - INFO - system - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - Temporary script location: /tmp/hellop5yj64ld
               solid = "hello"
    solid_definition = "hello"
            step_key = "hello.compute"
2020-04-03 13:29:16 - dagster - INFO - system - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - Running command: 
echo "Hello"
               solid = "hello"
    solid_definition = "hello"
            step_key = "hello.compute"
2020-04-03 13:29:16 - dagster - INFO - system - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - Hello

               solid = "hello"
    solid_definition = "hello"
            step_key = "hello.compute"
2020-04-03 13:29:16 - dagster - INFO - system - ed74cc0e-4ab8-4f0d-ad3f-3a0839c2ab3c - Command exited with return code 0
... you can see that it executes the solid, which is pretty undesirable. Am I doing something wrong?
n
hey Owen! I think you just need to change your code to something like:
Copy code
@pipeline
def my_pipeline():
    run_cmd_in_venv = bash_script_solid(
        '.../dagster_pipelines/hello_world.sh',
        name = 'hello'
    )
    run_cmd_in_venv()
execute_solid()
is an API primarily for running individual solids in a unit testing context; invoking the solid directly will add it to your pipeline
o
ah, okay. My bad. I'll try that. Thanks.
n
np! glad you flagged, we need to document this better; I filed https://github.com/dagster-io/dagster/issues/2345
👍 1
o
I'm looking for a way to dynamically pass variables through to a bash solid at runtime. I can see how to do it statically via the environment. However, I wasn't able to work out how to do it dynamically. I tried defining an input definition, but that didn't seem to work with the bash solids. Could be that I am doing something wrong, should the input defs work with bash solids?
n
@Owen yeah you may need to customize the solid impl, right now the default bash solids are generated w/ a single
Nothing
input: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-bash/dagster_bash/solids.py However, I just landed on master a change to extract the core bash solid compute functions to a separate utils module here: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-bash/dagster_bash/utils.py so I think you could define your own solid with something like:
Copy code
from dagster import solid, InputDefinition
from dagster_bash.utils import execute


@solid(input_defs=[InputDefinition('my_input', str)])
def my_solid(context, my_input):
    return execute('echo "{my_input}"'.format(my_input=my_input), output_logging='BUFFER', log=context.log)
(not tested, but that should work)
is that roughly what you’re trying to accomplish, binding the input arguments into the bash command somehow?
👍 1
o
Thanks for the feedback Nate, I'll give that a whirl.
Managed to carve out some time to try a little experiment. Seemed to work well. I'm hoping to be able to flip back onto this stuff shortly, so can then try it on a real-world example. But it seems very promising. Thanks Nate.
n
Sure thing! Keep us posted