Simon Späti
04/02/2020, 7:47 AMdelta_coordinate
’s and replace them in the delta_solid, other ideas are welcome 😊)
2. And wouldn’t it be a good idea to have the sql_solild function output as a `Materialization`(link) in that case as every delta table is persisted anyway? That way I have the possibility of the retry mechanism, which I otherwise wouldn’t have, correct? Or is that already integrated in the sql_solid function.
Thanks so much, maybe I can share my code when finished (if yes where?), that way might be possible to add it as example to dagster working with delta and spark and at the same time my newbie python code would be reviewed 😅alex
04/02/2020, 4:53 PMhave the sql_solild function output as a MaterializationSo
Output
s and Materialization
s are two different things. Outputs are dagster managed and allow things from solid to solid. A Materialization is just a structured event that says that some data was materialized somewhere that dagster does not manage.
So i could imagine given what you said that you may have Output
for a delta_coordinate
so down stream solids can interact with it, and you could yield Materialization
s to give more details about what was materialized in deltalake