https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
b

Binoy Shah

08/24/2022, 3:41 PM
One feedback: Please give Meltano Integration some “Love” - I am working towards integrating it myself, but there are so many Dagster secrets I havent uncovered yet, the work “appears” monumental, Cannot thank the engineering team and community enough, that’s helping me along the way, Justification/Rationalization: Meltano is another highly active AND mature “*Open Source ELT Tool*” that requires a versatile orchestrator like Dagster to do its Data Deeds, please Roadmap a build of dagster-meltano integration
s

sandy

08/24/2022, 6:02 PM
Thanks for the feedback. Here's an issue where we're tracking these requests: https://github.com/dagster-io/dagster/issues/4971. I linked this Slack thread from it
s

Stephen Bailey

08/26/2022, 10:58 AM
@Binoy Shah curious how you're doing this currently? i think the simplest way in my mind is to containerize your meltano project using ci/cd, then use the
kubernetes_job_op
to build custom schedules. one of the really hard parts here is managing the dependencies on the infra IMO. i've tried integrating a couple different ways and the kubernetes job way feels cleanest
b

Binoy Shah

09/01/2022, 2:31 PM
Its an interesting point @Stephen Bailey, my original idea was to kick off meltano and dbt as kube ops from a simplistic dagster pipeline, but it didnt give the feel of
Software Defined Assets
the feature which is a strong selling point of Daster.
Right now I am building Dagster + Meltano + dbt code artifacts in single docker image
and then trying to run it in kube
and letting the Kubernetes Job runner execute the ops defined from SDAs
s

Stephen Bailey

09/01/2022, 6:17 PM
the way i tend to think of it is:
Copy code
@asset
def meltano_elt_salesforce():
    # run meltano code
    return {"some": "metadata"}

@asset
def meltano_elt_google_sheets():
   # run meltano code
   return {"some": "metadata"}

@asset
def dbt_build_project(meltano_elt_salesforce, meltano_elt_google_sheets):
   # run dbt project code
   return {"some": "metadata"}
that is, the higher-level "meltano elt" job for each source is one asset, as is the
dbt build --select *
job. i think it's a bit hard to penetrate deeper into the job constructs (for example, mapping indvidual tables from each meltano job as assets)
yeah, and ify ou have everything in one image, the
# run meltnao code
is probably just
subprocess.run("meltano elt salesforce target-snowflake")
15 Views