I'm trying to generate daily reports and transformed subsets of tables over a large AWS Aurora MySQL database.
My plan so far is to use AWS DMS + AWS Glue with Hudi to create a daily snapshot of every tables.
Then I need to orchestrate all the queries and transforms to land a dataset per user.
Is Dagster the right tool to coordinate this? I'm new to the space and really confused with all the options 😕
03/22/2023, 11:03 PM
You can combine dbt + dagster. DBT will handle your transformations and dagster will orchestrate it. Check #dagster-dbt