https://dagster.io/ logo
#ask-community
Title
# ask-community
t

Thomas Mignon

12/15/2021, 8:25 PM
Hi 🙂 How can i pass a config yaml file and not directly from the code when Reusing an op definition with configured ? https://docs.dagster.io/concepts/configuration/configured#reusing-an-op-definition-with-configured From in the doc
Copy code
sample_dataset = configured(get_dataset, name="sample_dataset")({"is_sample": True})
To something like that
Copy code
sample_dataset = configured(get_dataset, name="sample_dataset")(context["is_sample"])
c

chris

12/15/2021, 8:27 PM
t

Thomas Mignon

12/15/2021, 9:01 PM
I don't think so, My problem is not clear sorry I need to pass to the second part of the configured function
Copy code
confgured()(hereisthesecondpart)
a configuration that i can write in the dagit interface as i do for example in
ops.template_runner_op.config
this is my code :
Copy code
from dagster import fs_io_manager, op, job, configured
from dagster_dask import dask_executor
from semaphore_scripts import template_runner
import os
from pathlib import Path

@op(
    config_schema={
        "command": str,
        "command_parameter": str,
        "semaphore_audience_version": str,
        "workspace": str
    },
    required_resource_keys={"io_manager"}
)
def template_runner_op(context):
    workspace: Path = Path(
        os.path.join(context.op_config["workspace"], f'{context.op_config["command"]}-{os.environ["PBS_JOBID"]}'))

    template_runner.TemplateRunner(
        command=context.op_config["command"],
        version=context.op_config["semaphore_audience_version"],
        command_parameter=context.op_config["command_parameter"],
        workspace=workspace,
        skip_signal=True
    ).run(exit=False)

purify = configured(template_runner_op, name="purify")(IDONTKNOW)
compact_purify = configured(template_runner_op, name="compact_purify")(IDONTKNOW)

@job(
    name='template_runner_job',
    resource_defs={"io_manager": fs_io_manager},
    executor_def=dask_executor
)
def template_runner_job():
    purify()
    compact_purify()
but i don't know what to write in the second part of configured for passing the context, in order to read config from the yaml in dagit via context.op_config
@chris maybe you have an idea ? 🙂
a

antonl

12/16/2021, 11:23 PM
I'm not sure I understand your usecase, but you should take a look at https://docs.dagster.io/_apidocs/config#dagster.configured , especially the example. In your case, your
IDONTKNOW
should be a callable that, when given the original yaml config, returns a new config that fits with the thing you're trying to configure. One common pattern is to hardcode values, but another is to do some sort of computation to adapt the source config. I would avoid trying to load a yaml file though in the configured call. The reason is that you don't necessarily know where the op will be executed, and so you'll have a hard time actually opening the file, in general. Maybe a better pattern is to treat loading the yaml as a separate op and then make a solid that takes a yaml file as an input?
c

chris

12/16/2021, 11:33 PM
I'm not sure I understand what you're trying to do here. Are you trying to pass a configuration value to configured that you get from a yaml file? Or are you trying to pass the path to the yaml as config?
t

Thomas Mignon

12/17/2021, 9:03 AM
I was trying to pass a configuration value to configured that you get from a yaml file but with the Factory Op pattern i found what i was searching for
Thnaks for the help guys 🙂
m

Manny Schneck

12/17/2021, 4:54 PM
Maybe take a look at
@config_mapping