Hi dagster-support! I would be so grateful if anyo...
# ask-community
n
Hi dagster-support! I would be so grateful if anyone could help me get a basic running example of my dbt project running on dagster! I got past the dependency issue by installing an older version of dagster-dbt, im no longer getting any errors on that! I still would love to talk to someone about setting up a basic running example of my DBT project running on dagster.
j
Hi Nicholas, have you gotten a chance to look at https://docs.dagster.io/integrations/dbt#using-dbt-with-dagster
n
Johann, I did, i guess im still trying to wrap my head around all the new concepts and get myself acquainted with dagster . it would be nice to chat with someone for a few minutes. sometimes talking it out helps me understand it better.
o
hi @Nicholas Buck! I can meet with you tomorrow if there's a convenient time -- there's also this example repo which might be helpful https://github.com/dagster-io/dagster/tree/master/examples/modern_data_stack_assets
❤️ 1
n
Thanks for your offer to help! I am on a time crunch and because of other obligations, I am pushed to try to get a working example today.
ill give this a shot, but i am not integrating with all the different toys at the moment like docker, snowflake etc
just trying to weed through and get only the most basic components running.
o
gotcha -- we're right in the thick of pushing out a major release for tomorrow, so I won't have time today. for what it's worth, you can basically delete any of the airbyte/pandas/etc. stuff from the example repo if you don't want to use those. you'll also be able to load that example without any modification, you just won't be able to kick off runs from dagit
n
my man! giving it a shot now.
stuck again. im almost there, but i get an error related to airbyte when i materialize all in dagit
o
hm if you want to circumvent airbyte, I believe you can modify constants.py so that
PG_DESTINATION_CONFIG = PG_SOURCE_CONFIG
, and then set
DBT_CONFIG = {"project_dir": DBT_PROJECT_DIR, "profiles_dir": DBT_PROFILES_DIR, "target": "skip_airbyte"}
you won't be able to hit materialize all but you can select the non-airbyte assets (cmd+click) and go from there
n
tried that but still gives airbyte-related errors - just ran the DBT and python
o
not at my computer right now, but do you mind sharing the error message?
n
sec i might have stumbled on the answer...
using pgadmin : in the database postgres_replica im not getting any tables appearing
heres where the errors start
2022-06-08 203501 -0400 - dagster - INFO - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - 2 of 3 START table model public.users_augmented................................. [RUN] 2022-06-08 203502 -0400 - dagster - ERROR - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - 1 of 3 ERROR creating table model public.orders_cleaned......................... [ERROR in 0.44s] 2022-06-08 203502 -0400 - dagster - ERROR - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - 2 of 3 ERROR creating table model public.users_augmented........................ [ERROR in 0.44s] 2022-06-08 203502 -0400 - dagster - INFO - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - 3 of 3 SKIP relation public.daily_order_summary................................. [SKIP] 2022-06-08 203502 -0400 - dagster - INFO - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - Finished running 3 table models in 3.30s. 2022-06-08 203502 -0400 - dagster - INFO - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - Completed with 2 errors and 0 warnings: 2022-06-08 203502 -0400 - dagster - ERROR - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - Database Error in model orders_cleaned (models\orders_cleaned.sql) 2022-06-08 203502 -0400 - dagster - ERROR - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - relation "public._airbyte_raw_orders" does not exist 2022-06-08 203502 -0400 - dagster - ERROR - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - LINE 11: from "postgres_replica"."public"."_airbyte_raw_orders" 2022-06-08 203502 -0400 - dagster - ERROR - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - ^ 2022-06-08 203502 -0400 - dagster - ERROR - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - compiled SQL at target\run\mds_dbt\models\orders_cleaned.sql 2022-06-08 203502 -0400 - dagster - ERROR - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - Database Error in model users_augmented (models\users_augmented.sql) 2022-06-08 203503 -0400 - dagster - ERROR - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - relation "public._airbyte_raw_users" does not exist 2022-06-08 203503 -0400 - dagster - ERROR - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - LINE 10: from "postgres_replica"."public"."_airbyte_raw_users" 2022-06-08 203503 -0400 - dagster - ERROR - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - ^ 2022-06-08 203503 -0400 - dagster - ERROR - resource:dbt - 7052dbfa-996b-41e3-82a2-2a022c79e551 - run_dbt_mds_dbt - compiled SQL at target\run\mds_dbt\models\users_augmented.sql
o
ah i see — sorry about that rabbit hole, i think airbyte would indeed need to work for this demo to function (at least without some more intensive modifications of the demo)
if you have a different dbt project that you can run locally, you can always toss that in instead
n
oh?
i have a local dbt project
o
the airbyte and postgres stuff is really just to get some sample data in
n
thats what i originally wanted to get running tbh
it uses our athena data
o
ah that makes sense
n
the dbt project works i know for sure
o
yeah if you just point load_assets_from_dbt_project at that and update DBT CONFIG in the constants it should work
n
trying to sell dagster intead of windows task scheduler for everything
😱 1
ill give it a try ty bro
o
nice hope it works!
n
crazy too i was so excited when i finally had docker and airbyte running
im sure its some dumb little thing