May I ask two conceptual questions all around sql_...
# announcements
May I ask two conceptual questions all around sql_solid and delta lake (
1. I’d like to have a delta-solid (same as sql_solid) where I can MERGE data frames into each other. Is it a good idea to extend the `materialization_strategy_output_types`with delta_table, or would you use a different function? The goal should also be, that the MERGE query is shown in dagster. (I’m not 100% sure yet how I’m dealing with table_names (don’t want them hard coded), but most probably I’m taking them from my
’s and replace them in the delta_solid, other ideas are welcome 😊) 2. And wouldn’t it be a good idea to have the sql_solild function output as a `Materialization`(link) in that case as every delta table is persisted anyway? That way I have the possibility of the retry mechanism, which I otherwise wouldn’t have, correct? Or is that already integrated in the sql_solid function. Thanks so much, maybe I can share my code when finished (if yes where?), that way might be possible to add it as example to dagster working with delta and spark and at the same time my newbie python code would be reviewed 😅
have the sql_solild function output as a Materialization
s and
s are two different things. Outputs are dagster managed and allow things from solid to solid. A Materialization is just a structured event that says that some data was materialized somewhere that dagster does not manage. So i could imagine given what you said that you may have
for a
so down stream solids can interact with it, and you could yield
s to give more details about what was materialized in deltalake
this is a pretty vague answer - but im not sure about the more specific questions without more context. If you send over your example code I’ld be happy to try to review it