some best practice questions for y all A very common pattern

Question

some best practice questions for y all A very common pattern for us is to execute a SQL query on Snowflake fetch the results and return the result set as a pandas DataFrame most of the time saving an artifact of the dataset in cloud storage as well Is there already an established pattern in Dagster for this Is it recommended to decompose this kind of pattern into multiple solids one for running the query one for constructing the DataFrame from the result set etc or rather just house this kind of operation in a single solid second question we need to use the result set mentioned above ^ and feed it into multiple downstream solids that can all be run in parallel These solids are largely identical and only differ by a small piece of config Is there a way to programmatically generate these solids with different config values and have them all be able to be run in parallel

max · Answer

hi < dwall>

dwall · Answer

wave

max · Answer

we are definitely seeing some people use factory functions to generate solids which might be an approach you could use for 2 alternatively you could use many aliases of the same solid and configure them explicitly in yaml etc which might be more explicit

max · Answer

the right answer to 1 probably depends on how reusable you want this pattern to be out of the box and what intermediate format you want to standardize on i suspect you will want a custom data type for your data frame and if you want your solids to speak data frames to each other that is if you ll never or rarely use the raw format you get back from Snowflake i would collapse the two operations into one solid

max · Answer

depending on how standard the process for saving the data frame to cloud storage is you might want to use intermediate storage <https dagster readthedocs io en 0 6 6 sections learn tutorial intermediates html>

max · Answer

this lets you write a serialization method once that will work with every one of your data frames and automagically gets you copies in cloud storage indexed by run id etc not without some overhead

max · Answer

if you need some subset of the artifacts to live in some particular place you might then have a specialized solid that writes the DF somewhere and emits a Materialization <https dagster readthedocs io en 0 6 6 sections learn tutorial materializations html>

dwall · Answer

cool this is really helpful thanks < max>