https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
a

Alexander Fife

02/23/2024, 4:37 PM
I love you Dagster, but your credit based on pricing materializations is absolutely killing me. I was thinking about setting up a new client on your stack, but this new pricing model is making me reconsider. I’ll have to do thousands of daily partitoned backfills to onboard them. • I shouldn’t have to pay 100 dollars in credits to backfill 7 years of daily data. Assuming that I can do that in a single asset. • Nor should I have to make a tradeoff between following good practices around partitioning to work around this. • Nor should I have to 24x my bill because I run a DAG hourly instead of daily. Especially on my own hardware with a hybrid deployment. Very Fivetranesque of you to make backfills something you have to be seriously careful about. I’ve seen this sentiment echoed on social media.
4
z

Zach

02/23/2024, 4:54 PM
Would running backfills as a single-run backfill still charge credits for each individual partition? I thought credits were consumed for each op / asset run
👌 1
r

Ravi Kotecha

02/23/2024, 5:33 PM
If you're worried about 100 bucks to backfill 7 years of data for a client you aren't charging anywhere near enough.
👎 1
a

Alexander Fife

02/23/2024, 6:34 PM
@Zach Not according to this github issue https://github.com/dagster-io/dagster/discussions/18963
z

Zach

02/23/2024, 6:43 PM
@Dagster Jarred can you clarify? You added "yes" to my comment, but the discussion @Alexander Fife linked seems to show that every partition materialization consumes its own credit, regardless of whether it was run as a single-run backfill
d

Dagster Jarred

02/23/2024, 6:44 PM
Sorry, I was clarifying that @alex is correct, currently single run backfills which generate partitions will cost credits per partition
z

Zach

02/23/2024, 6:45 PM
Oh re-read my comment and I see how the wording works with your "yes"
That's disappointing
d

Dagster Jarred

02/23/2024, 6:51 PM
Yeah, I can sympathize with the community here. I’ve talked to James from the GH issue about it too. Its on our to-do list to come up with some more customer friendly here, its just behind a set of other customer issues with wider impact. Sorry that our pricing model is causing friction for you @Alexander Fife, its a constant project to refine it
a

Alexander Fife

02/23/2024, 7:08 PM
I see the need to get paid for good work, which you guys do. Looking forward to see what happens in the future ✌️ Just sketches me out that if I had to backfill my client with 500 days of data, just to make a minor change, that would suddenly cost hundreds of dollars if they where on the new pricing. Also, as an agency, we were thinking about using Dagster for extracting marketing data for customers. 2 years of historical data from Facebook, ~8 different API calls, let’s say 3 ops/assets per API call. That’s 17520 assets. If we onboarded three customers in a month that would cost us 52 000 credits. Or if, we had 30 clients running on Dagster, and we found out we wanted to add a column to 4 of those API calls, that would be 261 000 credits to backfill. Or $7800. I don’t know. Maybe more transparency around what the “pro” version could help. Feels deeply unfair when everything is running on a hybrid executor as well.
d

Dagster Jarred

02/23/2024, 9:09 PM
There’s definitely a tension around wanting to have a billing mechanism that aligns with usage so that customers who engage with Dagster the most deeply are paying the most, and customers who are using it lightly have the smallest bills. The biggest challenge, as you’ve identified, is that some kinds of activities create big bills which aren’t necessarily super complex or indicate that a customer is deriving a lot of value from the process. A simple workaround for the backfill problem would be allowing customers to run a set number of backfills for free (say per month). @Alexander Fife happy to connect with you directly on DM if you have more questions, or to give you some background on Pro. Send me a message
3 Views