Is there a way to use

**kwargs

in ops? I have a few ops that are rendering Jinja2 template and need to pass information to the templates. It would be nice to be able to generalize the ops so that I can use them in other pipelines and just pass different info into the template, but if I try to pass keyword arguments in I get a

DagsterInvalidDefinitionError

.

dagster.core.errors.DagsterInvalidDefinitionError: Invalid dependencies: solid "update_templates" does not have input "latest_file". Available inputs: ['template_config', 'kwargs', 'start_after']

I’ve tried with an op like this:

@op(
    ins={
        "template_config": In(TemplateConfig),
        "kwargs": In(Optional[dict]),
        "start_after": In(Nothing),
    }
)
def update_temolates(
    context: OpExecutionContext,
    template_config: TemplateConfig,
    **kwargs,
):
    ...

In a graph similar to

@graph
def generate_datasets():
    dataset_path = download_dataset()
    template_config = template_and_destination()
    update_templates(template_config, dataset_path=dataset_path)

Hey @Alex Kerney - the closest thing we currently have to that is "fan-in", which allows ops to accept lists of arbitrary inputs (but not dicts): https://docs.dagster.io/concepts/ops-jobs-graphs/jobs-graphs#fixed-fan-in. @alex can correct me if I'm wrong.

Is there a way to use **kwargs in ops?

You technically can use

**kwargs

in your

@op

decorated function, but wiring the graph dependencies requires

ins

defined for each entry you want to connect to an upstream op. Sandys link above is a way to get around having to declare each input separately by having one input that gets fanned-in to and receives a list

I may be able to make that work, though a bit hacky. Something like:

class KeyValue:
    key: str
    value: Any

def to_key_value(key: str):
    @op(name=f"make_kv_{key}")
    def kv(value: Any):
        return KeyValue(key, value)

@graph
def generate_datasets():
    dataset_path = download_dataset()
    template_config = template_and_destination()
    dataset_path_kv = to_key_value("dataset_path")(dataset_path)
    update_templates(template_config, [dataset_path_kv])

+1 to Sandy's suggestion. One more thing that can be done is you can define an op inside an outer function and define **kwargs in outer function. Something like below

def outer function(**kwargs):
    @op(
        ins={
            "template_config": In(TemplateConfig),
            "kwargs": In(Optional[dict]),
            "start_after": In(Nothing),
        }
    )
    def update_templates(
        context: OpExecutionContext,
        template_config: TemplateConfig,
    ):
    #inside op
    #do something with **kwargs inside op

return update_templates