and I see Meltano here. Today it struck to me that there is yet void to be filled on how meltano will work under Dagster, when we are talking about Software Defined Assets (SDAs) becoming primary entities on Dagster platform
And yet, the video lists Meltano as one of the contender to work seamlessly with DBT under Dagster umbrella
Meltano does not have hooks for intercepting the
flow but Dagster, if it encompasses Meltano, can very well surface the SDAs for possibilities of variety of data flows
07/08/2022, 4:54 PM
Hi Binoy! Are you looking for a Meltano library integration for your workflow?
We’re tracking that feature with this GH issue. Feel free to upvote and follow along.
07/08/2022, 4:58 PM
@prha Not necessarily. I think a custom runner will do as long as it runs meltano’s CLI to execute meltano and record its outputs/Results into Data Assets. or something that captures Meltano based
E and L
I like the concept of SDA and its op based materializations, but doing EL via existing tools only makes it worthwhile if there are hooks for metadata exchange between the EL tool and Orchestrator (Meltano)