https://dagster.io/ logo
Title
k

Koby Kilimnik

10/24/2021, 4:11 PM
when i had modes, i just called one
prod
mode and the other
poc1
mode, each defined configuration and default env configuration for all of my resources, now it belongs to the repo which is a tad weirded abstractionwise
c

chris

10/24/2021, 6:49 PM
So the intended migration pattern is that each mode will correspond to a different job. That job will be able to define the resources, and the default configuration for those resources. As a result of the split, those jobs can be provided separately to different repos (whereas before each mode would live on the same pipeline), but I don't think I would say that the configuration belongs to the repo itself. As in, a pipeline with the following mode/preset combination:
@pipeline(
    mode_defs=[
        ModeDefinition(name="prod", resource_defs={"a": ...,    "b": ...})
    ]
)
def do_stuff():
    ...

my_preset = PresetDefinition(mode="prod", config={"resources": {"a": ... }})
Becomes a job like this:
@graph
def do_stuff():
    ...

do_stuff_job = do_stuff.to_job(name="do_stuff_prod", resource_defs={"a": ..., "b": ...}, config={"resources": {"a": ...}})
We can think of a pipeline as a computation graph with n different environments (1 for each mode). We can think of a job as a computation graph with a single environment. I think in some of our examples we show the to_job call happening within the repo definition, but that doesn't have to be the case. The job's definition is provided to the repo in the same way as the pipeline's definition is, but now it isn't enforced that all environments are provided to the same repository. Apologies for the essay. Does that help at all?
❤️ 1
d

dinya

10/25/2021, 5:45 AM
@chris what alternative is there now for PresetDefinition.from_files?
@chris I mean is it recommended to use the
yaml
package directly? Smth like
import yaml
from dagster import file_relative_path

cfg_file = open(file_relative_path(_file_, "default_config.yaml"))
default_cfg = yaml.full_load(cfg_file)

@graph
def do_stuff():
    ...

do_stuff_job = do_stuff.to_job(name="do_stuff_prod", resource_defs={"a": ..., "b": ...}, config=default_cfg)
or utils like
from dagster import config_from_files
k

Koby Kilimnik

10/25/2021, 7:10 AM
thanks chris, ill be missing modes 😞
was really helpful , so now i have a graph and when i turn it to a job i can pass it its configs
c

chris

10/25/2021, 3:04 PM
@dinya - that’s exactly right. The other thing to note is that the config arg on a job is only default config - can still be overridden
@Koby Kilimnik - that’s right
k

Koby Kilimnik

10/25/2021, 3:06 PM
😄