One feedback: Please give Meltano Integration some “Love” - I am working towards integrating it myself, but there are so many Dagster secrets I havent uncovered yet, the work “appears” monumental, Cannot thank the engineering team and community enough, that’s helping me along the way, Justification/Rationalization: Meltano is another highly active AND mature “*Open Source ELT Tool*” that requires a versatile orchestrator like Dagster to do its Data Deeds, please Roadmap a build of dagster-meltano integration

Thanks for the feedback. Here's an issue where we're tracking these requests: https://github.com/dagster-io/dagster/issues/4971. I linked this Slack thread from it

@Binoy Shah curious how you're doing this currently? i think the simplest way in my mind is to containerize your meltano project using ci/cd, then use the

kubernetes_job_op

to build custom schedules. one of the really hard parts here is managing the dependencies on the infra IMO. i've tried integrating a couple different ways and the kubernetes job way feels cleanest

Its an interesting point @Stephen Bailey, my original idea was to kick off meltano and dbt as kube ops from a simplistic dagster pipeline, but it didnt give the feel of

Software Defined Assets

the feature which is a strong selling point of Daster.

the way i tend to think of it is:

@asset
def meltano_elt_salesforce():
    # run meltano code
    return {"some": "metadata"}

@asset
def meltano_elt_google_sheets():
   # run meltano code
   return {"some": "metadata"}

@asset
def dbt_build_project(meltano_elt_salesforce, meltano_elt_google_sheets):
   # run dbt project code
   return {"some": "metadata"}

that is, the higher-level "meltano elt" job for each source is one asset, as is the

dbt build --select *

job. i think it's a bit hard to penetrate deeper into the job constructs (for example, mapping indvidual tables from each meltano job as assets)