I m trying to figure out how should I define a config schema dagster #announcements

I'm trying to figure out how should I define a con...

matas

08/02/2020, 3:24 PM

I'm trying to figure out how should I define a config_schema of this shape:

Copy code

categories:
  home:
  - {source: foo, mult: 1}
      
  business:
  - {source: bar, mult: 2}
  - {source: baz, mult: 3}

  ...

Here I have a dictionary with different keys (which I want to use as meaningfull values - home, business, etc) and an array of structured values (each one is {source: str, mult: int}) I see 3 ways now, all of them are quite cumbersome: • just use a Permissive for all the config - I would loose validation at the last leaf level, that's pitty • change the shape of config to have

Copy code

categories:
  - name: home
    sources:
    - {source: foo, mult: 1}
      
  - name: business
    sources:
    - {source: bar, mult: 2}
    - {source: baz, mult: 3}

  ...

looks ugly adding unnecessary 'name' and 'sources' • create my own DagsterType for the whole config - looks a little like overkill here. So am I missing something? Is there a better way or what should be more "dagsterish"? D

matas

08/02/2020, 3:45 PM

And BTW can I check if a supplied Array is not empty? Could it be added?

matas

08/02/2020, 9:21 PM

Oh, I've just understood that I can't use custom types in config_schemas. Is it deliberately or just not supported yet? For example I have my type "Formula" which is a plain text string that is parsed in a Formula class instance. I've made a DagsterType with loader and validation for it and now I want to insert it to my pipeline config as a part of a config_schema. Is it possible anyhow now? It doesn't look like an input for me

max

08/02/2020, 10:18 PM

I think you might be looking for

Selector

matas

08/02/2020, 10:28 PM

How could it help? These parts are not mutually exclusive

matas

08/02/2020, 10:35 PM

In fact I've chosen the 2nd option (reshaping) and it is quite ok for now (though it would still look better if my original shape could be supported). Anyway the more important question is about using a custom type inside a config_schema. In the example above I have Formula in each of this blocks, let it be

Copy code

categories:
  - name: home
    sources:
    - {source: foo, formula: '2*foo'}

  - name: business
    sources:
    - {source: bar, formula: 'bar**2'}
    - {source: baz, formula: 'baz+3'}

And I'd like to reuse loading\validation code for Formula type between my solids and pipelines schemas, so it looks naturally like a custom type. But when I try to use it as a part of config_shema I get a DagsterInvalidConfigDefinitionError

schrockn

08/02/2020, 11:51 PM

@matas if you want something close to what you said earlier:

Copy code

categories:
  home:
  - {source: foo, mult: 1}

  business:
  - {source: bar, mult: 2}
  - {source: baz, mult: 3}

  ...

I believe in order to get what you want you want a list of selectors. Each item in the list would be a dictionary with a single key

home

business

Copy code

categories:
  - home:
    - {source: foo, mult: 1}

  - business:
    - {source: bar, mult: 2}
    - {source: baz, mult: 3}

  ...

matas

08/03/2020, 7:27 AM

But I don’t know the whole set of available names: they are free to use. You would like to have an abracadabra-macaroni as a key, but it would be deprecated since there are only business and home presets

matas

08/04/2020, 11:49 AM

@schrockn @max and what about using a custom type inside a config_schema? what are the options here?

Open in Slack

Previous Next