Hello 👋, we are using Dagster Serverless, and we are having difficulties with the bandwidth limitation (Maximum of 100 GB of bandwidth per day).
We need to build a daily ETL process that involves downloading and uploading about 70GB of data each day.
The problem is that we will also have to backfill more than 2 years of historical data (about 60TB in total).
Is there a chance we could increase the bandwidth limit in order to perform the backfill using Dagster Serverless?
12/05/2022, 6:57 AM
this doesn't really answer your question, but I think [task compute] should be separated from [dagster instance], doing a computation from dagster instance can lead to OOM if it's not optimized. plus running tasks outside of dagster means you don't have to deal with scaling the dagster instance to handle more loads