https://dagster.io/ logo
Powered by
#announcements
Title
# announcements
d

dwall

01/24/2020, 8:33 PM
I'm looking to create a solid that is just an abstraction of another solid (adds a little bit of application-specific logic to bash command solid). Is the best way to do this to use composite solids? It feels like composite solids are really meant to be used to abstract sub-DAGs instead of single solids
a

alex

01/24/2020, 8:36 PM
there are cases where composite solid would make sense - but as you said its more so if your application-specific logic can be expressed as other solids you always want to happen around the core one (ie a sub-DAG)
I think this type of abstraction is probably something we need to figure out how to better support - I would be surprised if you dont end up having to fork the bash command solid to do what you want
depending on what application specific changes you want to add
d

dwall

01/24/2020, 8:40 PM
yeah - I think thats where Im hitting a roadblock
a

alex

01/24/2020, 8:41 PM
can you provide some more details about what you are trying to do?
d

dwall

01/24/2020, 8:41 PM
nothing crazy, honestly. Format the command with application-specific flags and set some env vars to pass into the bash command solid
a

alex

01/24/2020, 8:42 PM
it maybe the right thing to either add knobs to the existing bash command solid or refactor it to make subsets of it re-usable at the python level
oh! one thing you can do (which is poorly documented) is use the config mapping aspect of composite solids
that will at least solve the 
env
part of the problem
d

dwall

01/24/2020, 8:46 PM
okay right on. I think this makes sense
how would you go about doing some formatting of the command and then feeding it back into bash command solid?
a

alex

01/24/2020, 8:47 PM
lets see
you are passing the command in at the python call site to 
bash_command_solid
- could you wrap that in a function that does said processing for you (and potentially bundles it in a composte that locks the env vars)
d

dwall

01/24/2020, 8:50 PM
I think that is what Im trying to do currently, but I think I'm missing something
let me show you my solid definition
Copy code
@solid(
    description='A solid to invoke a ranch process.',
    input_defs=[InputDefinition(name='start', dagster_type=Nothing, description='Nothing.')],
    output_defs=[OutputDefinition(name='end', dagster_type=Nothing, description='Nothing.')],
    config={
        "command": Field(
            dagster_type=String,
            is_optional=False,
            description="The command to run"
        ),
        "app": Field(
            dagster_type=String,
            is_optional=False,
            description="The ranch app to run the command on."
        )
    }
)
def ranch_run(context):
    command = f"ranch run {context.solid_config['command']} --app {context.solid_config['app']}"
    solid = bash_command_solid(command)
a

alex

01/24/2020, 8:55 PM
ya 
bash_command_solid
is a solid factory function - so invoking it in a solid is going to try do that work at runtime
you can do it in a @pipeline or @composite_solid since those are evaluated at init time
honestly though - I think i would recommend looking at the impl of 
bash_command_solid
as a reference but just creating you’re own that allows you to do what you want
d

dwall

01/24/2020, 8:58 PM
oh okay, this is interesting. So I think I misunderstood the proper use of 
bash_command_solid
. Do y'all by chance have an example of how to properly leverage this solid?
a

alex

01/24/2020, 8:58 PM
its not like 
bash_command_solid
has a lot of battle testing on it - its really just a first stab at things
d

dwall

01/24/2020, 8:59 PM
fair enough. That may be what we end up doing
a

alex

01/24/2020, 8:59 PM
will happily accept a PR that breaks out any pieces you would like to re-use
have an example of how to properly leverage this solid?
the test cases are the only callsites we have
d

dwall

01/24/2020, 9:06 PM
ah okay Im getting it now
thanks @alex!
Powered by