Related to the above, given a config like the following, I'd like to express that either

path

is required, or

glue_database

and

glue_table

are required, but not both.

columns

is optional in either case. I've been looking at

Selector

for (

path

or (

glue_database

and

glue_table

)), but I can't think of a way to set this up. Is such a thing possible?

@input_hydration_config(
    Selector(
        {
            "parquet": Permissive({
                "path": Field(ReadPathType, is_required=True, description="Path to read from."),
                "glue_database": Field(String, is_required=True, description="Glue database to read from."),
                "glue_table": Field(String, is_required=True, description="Glue table to read from."),
                "columns": Field([String], is_required=False, description="Fields names to read in as columns."),
            }),
        }
    )
)

maybe something like this:

@input_hydration_config(
    Selector(
        {
            "parquet_file": {
                "path": Field(ReadPathType, is_required=True, description="Path to read from."),
                "columns": Field([String], is_required=False, description="Fields names to read in as columns."),
            },
            "parquet_glue": {
                "database": Field(String, is_required=True, description="Glue database to read from."),
                "table": Field(String, is_required=True, description="Glue table to read from."),
                "columns": Field([String], is_required=False, description="Fields names to read in as columns."),
            }),
        }
    )
)

Thanks! I originally wanted to avoid splitting it into two keys like

parquet_file

and

parquet_glue

, but it's probably the way to go.

ya you have to have the split somwhere - you could do a second layer of indirection if you wanted

parquet:
  file:
    path:
parquet:
  glue:
    database:

With that second layer direction idea, I think I'd run into the same problem I had originally. I'm not sure how I would write a config that says either

file

or

glue

have to be specified, along with a whole bunch of optional fields like

columns

under

parquet

. How would I form the

parquet

field in the config?

@input_hydration_config(
    Selector(
        {
            "parquet": Permissive({
                "???": Selector({
                	"file": …,
                	"glue": …,
                }),
                "columns": Field([String], is_required=False, description="Fields names to read in as columns."),
            }),
        }
    )
)

you cant (at least today) have sibling keys to a

Selector

so you would have to move

columns

in or add another layer:

@input_hydration_config(
    Selector(
        {
            "parquet": Permissive({
                "load_from": Selector({
                	"file": {...},
                	"glue": {...},
                }),
                "columns": Field([String], is_required=False, description="Fields names to read in as columns."),
            }),
        }
    )
)

and have

parquet:
  load_from:
    glue:
      database: ...
  columns: [...]

Ah, I hadn't considered moving them in another level like

load_from

. Thanks!