Hi all, maybe it is just basic question, but I cou...
# ask-community
Hi all, maybe it is just basic question, but I could not get my code working. I would like to have two ops that run shell command in order. I haven’t found a way to inject ins={“start”: In(Nothing)} directly to to create_shell_command_op directly, so I need to wrap them inside a factory. But in the inner op method, I could not get the command triggered because there is no context for invoking it. It is related to a previously posted question. Could some one help me?
Ok, found no way to reuse create_shell_command_op in the inner op. My workaround is to reuse the dagster_shell.utils.execute() method to have the execution done nicely as using the normal shell_op.
Sorry that I missed your ping. Yea I believe at this point, making a factory doesn’t help reduce the boilerplate so I think your workaround is what I’d recommend.
@yuhan Maybe could you please give me a hint, I am trying to grasp the concept of dependency in dagster. As long as an op consumes output of another one, so there is a dependency relation. But for most of built-in ops (e.g. create_shell_command_op, dataproc_op, etc… ), they don’t have this input definition. So user must either extend them or implement the logic from scratch. Is it correct or have I missed something? Thank you so much.
Hi Tung! underneath
it’s constructing an op with a Nothing input definition (source code) - this was built under the hypothesis that most of the shell command ops would not have upstream data dependency connected to them. You can read about Nothing type here and here’s an example with code snippet: https://docs.dagster.io/concepts/ops-jobs-graphs/jobs-graphs#order-based-dependencies-nothing-dependencies
Hi @yuhan, ok thanks for pointing out the start input for shell_op but how about other ops? Do you think is it a good Idea to provide a default “after” input for every op, that could be used for such situation? I think that could be done pretty easily by changing this line to: https://github.com/dagster-io/dagster/blob/2d219f4a405fba56bc3466f1165e5403c760e02b/python_modules/dagster/dagster/core/definitions/op_definition.py#L71
{"after": Nothing}.update({input_def.name: In.from_definition(input_def) for input_def in self.input_defs})
@sandy, @owen: I would like to hear your opinions about the default “after” input as well. The fact is, currently chaining op by order of execution in dagster is quite difficult (wrapping in group, another op def, etc.). With after, the entrypoint to new dagster user like me is much lowered. Sorry if I am not aware of any discussion about this before.
We've discussed this in the past and I recall they may have been an edge case where this breaks compatibility. @alex - do you recall off the top of your head?
The one thing I remember was breaking existing inputs on name collision. In the proposed change above,
is overridden by any users inputs with the same name which is one approach to solve for that. historical discussion around this on https://github.com/dagster-io/dagster/issues/4274
@sandy, @yuhan Based on the discussion in #4274 and the feature request in https://github.com/dagster-io/dagster/issues/7863 I think it is reasonable to add this ability to dagster op. IMHO it is the most intuitive way to define the op dependency.