https://dagster.io/ logo
k

King Chung Huang

06/27/2020, 7:52 PM
How do I declare a field with more than one value type? For example, I'm trying to write a
path
file that can either take a single string or a list of strings (corresponding to the
path
argument in
dask.dataframe.read_parquet
). Separately, the two value types can be written as:
Copy code
Field(String, …)
Field([String], …)
Is there something like a union type that can combine the two types?
m

max

06/27/2020, 9:07 PM
you may want a
Selector
k

King Chung Huang

06/28/2020, 2:46 AM
Can you elaborate on how a
Selector
would work in this case? Here's an excerpt of the original code.
Copy code
"parquet": Permissive({
    "path": Field(String, is_required=True, description="Path to read from."),
    "columns": Field([String], is_required=False, description="Fields names to read in as columns.")
}),
If
Selector
took a list of fields instead of a dict, I could imagine doing something like this.
Copy code
"parquet": Permissive({
    "path": Field(
        Selector([
            Field(String, is_required=True, description="Path to read from."),
            Field([String], is_required=True, description="List of paths to read from."),
        ]),
        is_required = True,
        description = "Path to read from."
    ),
    "columns": Field([String], is_required=False, description="Fields names to read in as columns.")
}),
Ultimately, I want the path to be valid for a string or a list of strings in its config.
Copy code
dataframe:
  parquet:
    path: /example/parquet
or
Copy code
dataframe:
  parquet:
    path:
      - /example/parquet/part0
      - /example/parquet/part1
Ah, I found https://github.com/dagster-io/dagster/issues/1697 about adding a union type. The Slack message linked in #1718 has an example of a custom
DagsterType
to handle a list of types.
4 Views