I love you Dagster, but your credit based on pricing materializations is absolutely killing me. I was thinking about setting up a new client on your stack, but this new pricing model is making me reconsider. I’ll have to do thousands of daily partitoned backfills to onboard them. • I shouldn’t have to pay 100 dollars in credits to backfill 7 years of daily data. Assuming that I can do that in a single asset. • Nor should I have to make a tradeoff between following good practices around partitioning to work around this. • Nor should I have to 24x my bill because I run a DAG hourly instead of daily. Especially on my own hardware with a hybrid deployment. Very Fivetranesque of you to make backfills something you have to be seriously careful about. I’ve seen this sentiment echoed on social media.

Would running backfills as a single-run backfill still charge credits for each individual partition? I thought credits were consumed for each op / asset run

If you're worried about 100 bucks to backfill 7 years of data for a client you aren't charging anywhere near enough.

@Zach Not according to this github issue https://github.com/dagster-io/dagster/discussions/18963

@Dagster Jarred can you clarify? You added "yes" to my comment, but the discussion @Alexander Fife linked seems to show that every partition materialization consumes its own credit, regardless of whether it was run as a single-run backfill

Sorry, I was clarifying that @alex is correct, currently single run backfills which generate partitions will cost credits per partition

Oh re-read my comment and I see how the wording works with your "yes"

Yeah, I can sympathize with the community here. I’ve talked to James from the GH issue about it too. Its on our to-do list to come up with some more customer friendly here, its just behind a set of other customer issues with wider impact. Sorry that our pricing model is causing friction for you @Alexander Fife, its a constant project to refine it

I see the need to get paid for good work, which you guys do. Looking forward to see what happens in the future ✌️ Just sketches me out that if I had to backfill my client with 500 days of data, just to make a minor change, that would suddenly cost hundreds of dollars if they where on the new pricing. Also, as an agency, we were thinking about using Dagster for extracting marketing data for customers. 2 years of historical data from Facebook, ~8 different API calls, let’s say 3 ops/assets per API call. That’s 17520 assets. If we onboarded three customers in a month that would cost us 52 000 credits. Or if, we had 30 clients running on Dagster, and we found out we wanted to add a column to 4 of those API calls, that would be 261 000 credits to backfill. Or $7800. I don’t know. Maybe more transparency around what the “pro” version could help. Feels deeply unfair when everything is running on a hybrid executor as well.

There’s definitely a tension around wanting to have a billing mechanism that aligns with usage so that customers who engage with Dagster the most deeply are paying the most, and customers who are using it lightly have the smallest bills. The biggest challenge, as you’ve identified, is that some kinds of activities create big bills which aren’t necessarily super complex or indicate that a customer is deriving a lot of value from the process. A simple workaround for the backfill problem would be allowing customers to run a set number of backfills for free (say per month). @Alexander Fife happy to connect with you directly on DM if you have more questions, or to give you some background on Pro. Send me a message