https://dagster.io/ logo
#dagster-cloud
Title
# dagster-cloud
b

Brandon Peebles

08/23/2023, 6:13 PM
If we were to set up a hybrid architecture with ECS and Dagster to run dbt, would the logs from dbt still be accessible from within Dagster? Or would those be confined to CloudWatch?
I'm also debating MWAA and my concern with either platform is whether we'll be able to centralize logging back into the orchestrator. Would prefer not to hop all over the place to view the logs for my tasks 🙂
b

ben

08/23/2023, 6:30 PM
Hi Brandon, yes, the Cloud agent forwards logs (both raw stdout/stderr and structured logs) to the orchestrator, so you can view them in the web UI.
🙌 1
This includes the dbt logs from materializing dbt models
b

Brandon Peebles

08/23/2023, 6:34 PM
Nice! Is that just on by default or is there extra work to configure that? Also, as I'm trying to help my team evaluate Dagster vs Airflow, do you know if this is also feasible on MWAA? And if it is, is it simple to do or more complex? I assume we'd still be triggering ECS tasks just like Dagster would so my guess is yes, but I'm curious to know if this is a clear issue with Airflow that Dagster has solved
👍 1
b

ben

08/24/2023, 10:44 PM
Yeah, it’s on by default (though there are ways to opt out for folks who don’t want to send logs)
p

Pierce Few

08/24/2023, 11:02 PM
@Sean Lopp
s

Sean Lopp

08/24/2023, 11:05 PM
For MWAA the log experience is tied to the operators implementation But often that means cloudwatch You also loose the asset view there
b

Brandon Peebles

08/25/2023, 12:24 AM
Thanks all!
21 Views