Thomas Mignon
12/15/2021, 8:25 PMsample_dataset = configured(get_dataset, name="sample_dataset")({"is_sample": True})
To something like that
sample_dataset = configured(get_dataset, name="sample_dataset")(context["is_sample"])
chris
12/15/2021, 8:27 PMThomas Mignon
12/15/2021, 9:01 PMconfgured()(hereisthesecondpart)
a configuration that i can write in the dagit interface as i do for example in ops.template_runner_op.config
from dagster import fs_io_manager, op, job, configured
from dagster_dask import dask_executor
from semaphore_scripts import template_runner
import os
from pathlib import Path
@op(
config_schema={
"command": str,
"command_parameter": str,
"semaphore_audience_version": str,
"workspace": str
},
required_resource_keys={"io_manager"}
)
def template_runner_op(context):
workspace: Path = Path(
os.path.join(context.op_config["workspace"], f'{context.op_config["command"]}-{os.environ["PBS_JOBID"]}'))
template_runner.TemplateRunner(
command=context.op_config["command"],
version=context.op_config["semaphore_audience_version"],
command_parameter=context.op_config["command_parameter"],
workspace=workspace,
skip_signal=True
).run(exit=False)
purify = configured(template_runner_op, name="purify")(IDONTKNOW)
compact_purify = configured(template_runner_op, name="compact_purify")(IDONTKNOW)
@job(
name='template_runner_job',
resource_defs={"io_manager": fs_io_manager},
executor_def=dask_executor
)
def template_runner_job():
purify()
compact_purify()
but i don't know what to write in the second part of configured for passing the context, in order to read config from the yaml in dagit via context.op_configantonl
12/16/2021, 11:23 PMIDONTKNOW
should be a callable that, when given the original yaml config, returns a new config that fits with the thing you're trying to configure.
One common pattern is to hardcode values, but another is to do some sort of computation to adapt the source config. I would avoid trying to load a yaml file though in the configured call. The reason is that you don't necessarily know where the op will be executed, and so you'll have a hard time actually opening the file, in general. Maybe a better pattern is to treat loading the yaml as a separate op and then make a solid that takes a yaml file as an input?chris
12/16/2021, 11:33 PMThomas Mignon
12/17/2021, 9:03 AMManny Schneck
12/17/2021, 4:54 PM@config_mapping