I'm playing around with creating a csv reader like the one shown in the Dagster tutorial except using the pandas.read_csv function. In the "Parametrizing solids with config" of the tutorial here: https://dagster.readthedocs.io/en/0.7.0/sections/tutorial/config.html it explains:

We obviously don't want to have to write a separate solid for each permutation of these parameters that we use in our pipelines – especially because, in more realistic cases like configuring a Spark job or even parametrizing the read_csv function from a popular package like Pandas, we might have dozens or hundreds of parameters like these.

But hoisting all of these parameters into the signature of the solid function as inputs isn't the right answer either:

...

The solution is to define a config schema for our solid
@solid(
    config={
        'delimiter': Field(
            String,
            default_value=',',
            is_required=False,
            description=('A one-character string used to separate fields.'),
        ),
        'doublequote': Field(
            Bool,
            default_value=False,
            is_required=False,
            description=(
                'Controls how instances of quotechar appearing inside a field '
                'should themselves be quoted. When True, the character is '
                'doubled. When False, the escapechar is used as a prefix to '
                'the quotechar.'
            ),
        ),
        'escapechar': Field(
            String,
            default_value='\\',
            is_required=False,
            description=(
                'On reading, the escapechar removes any special meaning from '
                'the following character.'
            ),
        ),
        'quotechar': Field(
            String,
            default_value='"',
            is_required=False,
            description=(
                'A one-character string used to quote fields containing '
                'special characters, such as the delimiter or quotechar, '
                'or which contain new-line characters.'
            ),
        ),
        'quoting': Field(
            Int,
            default_value=csv.QUOTE_MINIMAL,
            is_required=False,
            description=(
                'Controls when quotes should be generated by the writer and '
                'recognised by the reader. It can take on any of the '
                'csv.QUOTE_* constants'
            ),
        ),
        'skipinitialspace': Field(
            Bool,
            default_value=False,
            is_required=False,
            description=(
                'When True, whitespace immediately following the delimiter '
                'is ignored. The default is False.'
            ),
        ),
        'strict': Field(
            Bool,
            default_value=False,
            is_required=False,
            description=('When True, raise exception on bad CSV input.'),
        ),
    }
)
def read_csv(context, csv_path: str):
    with open(csv_path, 'r') as fd:
        lines = [
            row
            for row in csv.DictReader(
                fd,
                delimiter=context.solid_config['delimiter'],
                doublequote=context.solid_config['doublequote'],
                escapechar=context.solid_config['escapechar'],
                quotechar=context.solid_config['quotechar'],
                quoting=context.solid_config['quoting'],
                skipinitialspace=context.solid_config['skipinitialspace'],
                strict=context.solid_config['strict'],
            )
        ]

    <http://context.log.info|context.log.info>('Read {n_lines} lines'.format(n_lines=len(lines)))

    return lines

This example seems to work great but what about parameters with multiple types ? For example, the

index_col

keyword arg of the pandas read_csv that has multiple types ?https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html

index_colint, str, sequence of int / str, or False, default None

Is using the Any type really appropriate for these cases?

yep, i was looking for a good familiar example of a function with too many parameters 😉

This was discussed previously that this approch doesn't seem any better since you would still have to duplicate all the arguments in the nested function. But that aside, I was curious about how to handle arguments with multiple types like many of the pandas keywords for read_csv and read_excel.

it's true that writing the strongly typed schema is a bit of a hassle, but then the library solid can be reused however many times -- if you're reading a csv more than once, it's probably overall a win

agreed. and I completely understand the need for having to define them in the first place but I'm curious how this would play out if like it mentioned in the tutorial you had the task of having to replicate all of the spark APIs and it's parameters

in practice, we did that programmatically https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-spark/dagster_spark/configs_spark.py

interesting. could the same be applied to arbitrary functions to turn them into solids with a wrapper ?

definitely -- we have seen some proposals to do that and expect that people will write tools for this purpose

yep, basically this ^ is exactly what I was getting at.

narrowly, for this particular problem, you can either call those params

Any

, or we do have the

Selector

facility

so like you mentioned if you wanted to be specific perhaps the

Selector

would help out, or the

Any

type.

id be interested in how, in a perfect world, youd like to be able to represent that multiple typing

ya, ok. apologies for the duplicitive and lengthy post. most just thinking out loud.

yep np

this post is great

many eyes

exactly the type of thing we want to hash out with user feedback and questions

Something like this would be great from an end user experience. Obviously, a bit more involved implementation wise.

'index_col': Field(
            [int, str, Array],
            default_value=(1, '1', []),
            is_required=False,
            description=(
                'Column(s) to use as the row labels of the DataFrame, either given as string name or column index. If a sequence of int / str is given, a MultiIndex is used.'),
),

that’s great to hear