https://dagster.io/ logo
Join Slack
Powered by
I'm looking to create a solid that is just an abst...
# announcements
d
I'm looking to create a solid that is just an abstraction of another solid (adds a little bit of application-specific logic to bash command solid). Is the best way to do this to use composite solids? It feels like composite solids are really meant to be used to abstract sub-DAGs instead of single solids
a
there are cases where composite solid would make sense - but as you said its more so if your application-specific logic can be expressed as other solids you always want to happen around the core one (ie a sub-DAG)
I think this type of abstraction is probably something we need to figure out how to better support - I would be surprised if you dont end up having to fork the bash command solid to do what you want
depending on what application specific changes you want to add
d
yeah - I think thats where Im hitting a roadblock
a
can you provide some more details about what you are trying to do?
d
nothing crazy, honestly. Format the command with application-specific flags and set some env vars to pass into the bash command solid
a
it maybe the right thing to either add knobs to the existing bash command solid or refactor it to make subsets of it re-usable at the python level
oh! one thing you can do (which is poorly documented) is use the config mapping aspect of composite solids
that will at least solve the 
env
part of the problem
d
okay right on. I think this makes sense
how would you go about doing some formatting of the command and then feeding it back into bash command solid?
a
lets see
you are passing the command in at the python call site to 
bash_command_solid
- could you wrap that in a function that does said processing for you (and potentially bundles it in a composte that locks the env vars)
d
I think that is what Im trying to do currently, but I think I'm missing something
let me show you my solid definition
Copy code
@solid(
    description='A solid to invoke a ranch process.',
    input_defs=[InputDefinition(name='start', dagster_type=Nothing, description='Nothing.')],
    output_defs=[OutputDefinition(name='end', dagster_type=Nothing, description='Nothing.')],
    config={
        "command": Field(
            dagster_type=String,
            is_optional=False,
            description="The command to run"
        ),
        "app": Field(
            dagster_type=String,
            is_optional=False,
            description="The ranch app to run the command on."
        )
    }
)
def ranch_run(context):
    command = f"ranch run {context.solid_config['command']} --app {context.solid_config['app']}"
    solid = bash_command_solid(command)
a
ya 
bash_command_solid
is a solid factory function - so invoking it in a solid is going to try do that work at runtime
you can do it in a @pipeline or @composite_solid since those are evaluated at init time
honestly though - I think i would recommend looking at the impl of 
bash_command_solid
as a reference but just creating you’re own that allows you to do what you want
d
oh okay, this is interesting. So I think I misunderstood the proper use of 
bash_command_solid
. Do y'all by chance have an example of how to properly leverage this solid?
a
its not like 
bash_command_solid
has a lot of battle testing on it - its really just a first stab at things
d
fair enough. That may be what we end up doing
a
will happily accept a PR that breaks out any pieces you would like to re-use
have an example of how to properly leverage this solid?
the test cases are the only callsites we have
d
ah okay Im getting it now
thanks @alex!
Open in Slack
PreviousNext
Powered by