https://dagster.io/ logo
e

Eric

11/22/2019, 1:03 AM
Hi All, I'm going through the tutorial and have a question. In the section "Parametrizing solids with config" it explains the difference between inputs and configs. The "don't do it like this it's bad practice" example it uses says not to hoist all the parameters of the to_csv function into the arguments of the solid. I completely agree that having to rewrite all the permutations of arguments for all the underlying function(s) calls would be a nightmare. However, the "proper" solution putting them into the config seems it's doing exactly that. Would it not be easier (specifically easier not necessarily better since I understand the point of defining the config options) to just have a parameter of **kwargs to be passed in and avoid having to redefine options in the functions being used ?
m

max

11/22/2019, 1:10 AM
Hi Eric, you could certainly use what we call a PermissiveDict in config, or just accept a dict input, and pass that through to the underlying python function, but in doing so you'll lose all the advantages of type-checking in dagit, etc.
We are considering an API like
as_solid(python_fn)
which would automagically define all of the config parameters by introspecting the kwargs of the underlying python function, for cases like this where you want to wrap some existing library
You may also be interested in looking at composite solids, which provide the basic facility required to curry/partially evaluate solids with loads of config parameters
e

Eric

11/22/2019, 1:19 AM
Thanks for the response max. I agree it doesn't feel right to use the permissivedict but it's good to know that's an option. Thank you for answering my next question too of would it be possible to make an existing function as a solid. Is there a timeline for something like that or just in the works ?
m

max

11/22/2019, 1:43 AM
if it's something that would be useful to you, i think it would be pretty straightforward for us to implement
my thoughts were that it might be nice to be able to autowrap big chunks of the python standard library using it
e

Eric

11/22/2019, 4:17 AM
That would be great for our use case and I'm sure many others. Most of our pipelines are relatively small and consist mostly of shuffling data from one database to another with a few simple transformations. The majority of these pipelines also utilize pandas and dataframes. Having the ability to quickly wrap existing functions without having to redefine parameters and still get all the great benefits of dagster would make development even better ! If that's something on your road map I look forward to keeping an eye out for future updates.
👍 1