King Chung Huang
06/27/2020, 7:52 PMpath
file that can either take a single string or a list of strings (corresponding to the path
argument in dask.dataframe.read_parquet
). Separately, the two value types can be written as:
Field(String, …)
Field([String], …)
Is there something like a union type that can combine the two types?max
06/27/2020, 9:07 PMSelector
King Chung Huang
06/28/2020, 2:46 AMSelector
would work in this case? Here's an excerpt of the original code.
"parquet": Permissive({
"path": Field(String, is_required=True, description="Path to read from."),
"columns": Field([String], is_required=False, description="Fields names to read in as columns.")
}),
If Selector
took a list of fields instead of a dict, I could imagine doing something like this.
"parquet": Permissive({
"path": Field(
Selector([
Field(String, is_required=True, description="Path to read from."),
Field([String], is_required=True, description="List of paths to read from."),
]),
is_required = True,
description = "Path to read from."
),
"columns": Field([String], is_required=False, description="Fields names to read in as columns.")
}),
Ultimately, I want the path to be valid for a string or a list of strings in its config.
dataframe:
parquet:
path: /example/parquet
or
dataframe:
parquet:
path:
- /example/parquet/part0
- /example/parquet/part1
DagsterType
to handle a list of types.