19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
hey folks!
we’re heavy dbt users running several ...
# integration-dbtu
0xDoing _
07/01/2023, 4:14 PMhey folks!
we’re heavy dbt users running several 10+ builds per hour, and we want to move our orchestration layer to dagster
we’re currently trying to figure out what’s the best dagster deployment for us
we thought about using dagster cloud, but our understanding is that a dbt build would consume serverless workers that have much more resources than what we actually need in order to run dbt
how do y’all recommend handling dbt-focused dagster deployments?
r
rex
07/03/2023, 2:22 PMWe’d still recommend that you use Dagster Cloud so that you don’t have to worry about managing the infrastructure for the orchestration layer.
Re: your question about serverless workers, this seems to be a pricing question: you can check our website for details on that. You are correct that in the case for dbt, the worker is just shuffling compute, which is all happening in the data warehouse layer.
u
0xDoing _
07/03/2023, 2:23 PMdo you have a sense of how many workers 1 dbt build uses? is it just 1 worker?
r
rex
07/03/2023, 2:25 PMJust to be clear, in Serverless, we don’t charge by worker number. We charge by compute minute.
But yes, a
dbt buildu
0xDoing _
07/03/2023, 2:26 PMgotcha!
we don’t charge by worker number. We charge by compute minute.but if I’m doing something heavy I’ll need more than one process/worker, so I’ll be paying for more compute, right?
r
rex
07/03/2023, 2:29 PMYeah — if your future computations translate into increased step duration for your Dagster jobs, then yes you will be paying for that.
u
0xDoing _
07/03/2023, 2:30 PMsounds good!
f
Fraser Marlow
07/04/2023, 7:20 PMHey Ricardo. Our pricing model actually just changed due to use feedback. You can read the details here.
The punchline is that we have dropped the existing compute-duration based pricing mechanism in favor of event-based pricing (called Dagster Credits) and per-seat pricing above certain limits.
Fraser Marlow
07/04/2023, 7:20 PMSorry for the timing, but this happened just last week.
u
0xDoing _
07/04/2023, 11:04 PMhey @Fraser Marlow! thanks for letting me know
0xDoing _
07/04/2023, 11:06 PMunfortunately I think this would be even more expensive for us… we have thousands of dbt models that are materialized every hour and my understanding is that each 1k dbt models materialized hourly would cost 2.4k/month with the new pricing model 😞
and this would be only for orchestration, the real compute is made by snowflake which already costs us dozens of thousands of dollars
👍 4
➕ 3
j
Joseph Florencio
07/05/2023, 5:41 PMI agree with @0xDoing _ that this new pricing doesn't work well for DBT based projects. We use DBT with BigQuery and have about ~1.2k models we materialize several times a day. In the past 30 days, we materialized 174,991 models. Ignoring op and seat costs, Dagster credits for the Team plan would cost us (174,991-10,000)*$0.03 + $100 = $5,049.73. For reference, our entire BigQuery bill last month was $11,129.41. We currently materialize these models in one of our k8s clusters with 2Gi and 200 millicores, so about $10/mo worth of VM compute on GCP with n2 machine types.
👀 2
Joseph Florencio
07/05/2023, 5:43 PMNot that Dagster pricing should be crazy low as it does provide a lot of value which we're sold on, but it's difficult to justify the cost of Dagster cloud with finance when it's ~45% of our current DWH spend.
f
Fraser Marlow
07/05/2023, 5:58 PMThanks for the feedback and the context. This is really helpful as we think this through.
👍🏽 1
👍 1
s
sandy
07/05/2023, 11:54 PMQuestion for you @0xDoing _ and @Joseph Florencio: what % of your models are views?
We're investigating making changes that would make it easy to avoid running models that are views if the models haven't changed since the last time they ran. Depending on your setup, that might be able to trim costs significantly?
u
0xDoing _
07/06/2023, 12:00 AMhey @sandy! I would say like 40% of the models are views or ephemeral
that would help with the costs but 60% of several thousand bucks is still a lot
j
Joseph Florencio
07/06/2023, 12:45 AM@sandy out of those materializations, 73,262 are views. So ~42%.
Joseph Florencio
07/06/2023, 2:53 AMFull disclosure: We do not use Dagster in production yet. I’ve prototyped a migration for my company, and I’ll be doing the migration sometime this quarter.
My 2 cents: I think the challenge is that not all materializations are equal. A company with a single data analyst can easily create hundreds of DBT models in a short time, many of which will be trivial - essentially mirrors of data loaded from providers like Segment/Rudderstack/Fivetran/etc. On the other side you have pipelines that are written in frameworks like Airflow that are substantially more difficult to write, require some amount of compute, and you’ll typically have far less of. This is super reasonable pricing for our airflow style jobs. It doesn’t feel as reasonable for DBT.
I’ve been debating between Dagster OSS vs Cloud Hybrid. In the past 30 days we spent 12,091 minutes invoking DBT, which for the previous Dagster pricing I assume would map to 12,091 * $0.03 = $362.73/mo, or ~$4.35k/yr. This new pricing would make it $60k/yr without view optimizations or $36k/yr with optimizations just for DBT invocations. For smaller companies like mine, this essentially takes the cloud offering off of the table. Compared to SaaS pricing for GitLab, GitHub, Astronomer, GCP Cloud Composer, DBT, and others, this feels like you’re double paying - once for the premium per developer seat pricing and then a second time for these tiny materializations even when you bring the compute.
FWIW, I can completely understand why you guys came up with this new pricing model, and it likely makes sense in a lot of cases. I don’t have a proposal for how you could improve the pricing that can’t be abused so it doesn’t penalize DBT users so much. Just wanted to relay my feedback.
👍 2
➕ 2
g
Gabriel Montañola
07/06/2023, 2:54 PMI'm on the same boat, plenty of dbt materializations and little compute time.
We'll try to trim our pipelines to avoid useless materializations but I'm not sure if it will help significantly.
On a perfect world my business users wouldn't mind daily
dbt runsd
Dagster Jarred
07/06/2023, 3:56 PMhey everyone, just wanted to acknowledge the feedback. Thanks for taking the time to share with us, we’re reading everything and discussing.
daggy love 3
☝️ 1
h
Hoang Lam
07/07/2023, 4:47 AMHi everyone,
I'm exploring with the new pricing model using event-based.
@Dagster Jarred and @Fraser Marlow
In my project, I use dynamic mapping and collect to run my ops. (https://docs.dagster.io/_apidocs/dynamic)
How does Dagster count credit in the case I use mapping and collect?
d
Dagster Jarred
07/07/2023, 6:48 AMhey hoang! this would be a great question for the primary thread since other people might also ask it, but the map step itself clones an op for each separate dynamic out, and each executed op there would consume 1 credit
👍 1
h
Hoang Lam
07/07/2023, 6:50 AMThank you @Dagster Jarred for confirming it.
y
Yaron Levi
07/14/2023, 4:10 AM@Dagster Jarred I think we are in the same situation regarding the new Pricing.
Currently we use Prefect Cloud, but looking for an alternative since we don’t have the dev time to manage the infra which is a little bit flakey. (ECS Tasks).
We have an ETL job from Mongo to Snowflake.
It runs every 15 minutes and syncs around 80 collections to Snowflake.
We want to use a single @job that runs 80 @ops.
I assume the @ops will be good here since it allows parallel execution.
Let’s say that in 1 hour the job will run 3 times.
So calculating the number of ops per month:
3 * 80 * 730 = 175200 ops
Which would cost * $0.03 = $5256 per month.
In Prefect this would be super cheap. As you are not billed per Task. (Task is like an Op)
(even if we cut down to sync each 1 hour, we are at $1752 per month)
Am I missing something here regarding the use-case and my calculations?
👍 1
d
Dagster Jarred
07/14/2023, 4:13 AMhey @Yaron Levi let me follow up with you on DM
👍 1
h
Hoang Lam
07/14/2023, 5:30 AMYeah, I think with the new pricing of Dagster Cloud, the cost is very expensive compared to other ones (Prefect)
d
Dagster Jarred
07/14/2023, 4:04 PMgoing to follow up with you on DM
Dagster Jarred
08/17/2023, 6:05 AMhey friends, not sure if this will be hugely impactful for anyone on this thread who is probably generating what we would consider ‘enterprise’ levels of activity, but we’ve just expanded the number of credits included in the solo and team plans:
Solo -> 7,500 credits included per month
Team -> 30,000 credits included per month
This hasn’t been announced but we rolled out the actual price changes today, and the pricing page itself will change tomorrow
4 Views