I have 2 ops that require inputs. How do I pass that through the config in job? I tried

production_job = graphname.to_job(
   config= {"ops": {"op1":{"inputs":{"arg1":"bla", "ar2":"bla"}},
                    {"op2":{"inputs":{"arg1: "something"}
            })

But that doesn't work. It does work for 1 op, but I get

unexpected config entry

. I tried a list in stead of 2 dictionaries. But that is also not allowed. What is the best way to pass the arguments to the ops from the job?

are your inputs literally named

arg1

and

arg2

?

I changed the ops to take these configuration settings from the context.

config = {'ops':{'op1':{'config':{'arg1':'arg'}},{'op2':{'config':{'arg2':'arg'}}

I am able to supply these things with a yaml config in dagit. So something is wrong

I use a yaml file as well and have multiple arguments without issue

Can I use yaml files as configuration for schedules?

Haven’t tried. I don’t know if dagster natively has a way of configuring a schedule via yaml, but if you’re comfortable using a yaml loader, it shouldn’t be too difficult

My usecase is this: • operator 1: read a specific file from S3, pass the file along to another operator • Do something with that file, pass that file along to • operator3: write the resulting file to another S3 location. I want to specify for operator 1 where the file lives on S3, and for operator 3 where to write the file.

I’m doing the same, except with Google Storage 🙂 looks something like this:

ops:
  load_data:
    inputs:
      file_paths: 
        - "source/path/example"
  save_to_lake:
    inputs:
      project: myproject
      table: example_table

Yeah that works in dagit. But I can't put it in the job definition

I load my configs using

from dagster.utils.yaml_utils import load_yaml_from_path

and then just point the job definition to the loaded yaml, seems to work reasonably enough

Thank Alex! I will try some of this out. Looks a lot like how I used to configure airflow jobs!

no problem! Hope it’s helpful 🙂

ah too bad, loading in the yaml config with that tool leads to a failing python dictionary

it fails on load? might wanna double-check that yaml format then; it’s just using

yaml.safe_load

under the hood:

def load_yaml_from_path(path):
    check.str_param(path, "path")
    with open(path, "r") as ff:
        return yaml.safe_load(ff)

So this yaml is valid when supplied in dagit and will also run a job:

ops:
  ops1:
    config:
      filename: file1.csv
      filepath: dagster-test/file1.csv
  ops2:
    config:
      filepath: dagster-test/output/file2.csv

However that yaml transformed (with load yaml from path) into a dictionary is invalid and dagit refuses to load it.

{'ops' {'ops1': {'config': {'filename': 'file1.csv', 'filepath':'filepath'}}, 'ops2':{'config':{'filepath':'filepath2'}} }}

not an idiot, we’ve all been there 🙂