Hey @Joe Im back! We needed to increase the frequency that our largest sensor ran every 10hours to every 10 minutes and thats brought back the timeout issues (ss below) This has led me to conclude that we should not trigger job runs inside a sensor. The sensor just doesnt seem to have the resources to handle that. What im considering doing is changing the sensor to 1. Only add partitions 2. Adding automaterialization to process the partitions Im taking that approach in a draft pr here: https://github.com/airbytehq/airbyte/pull/28190 Im also considering going one step further to changing the system to 1. The sensor notices new file and triggers a job 2. The job is responsible for adding the new partitions 3. The new partitions are then processed via auto materialize My direct questions for you are 1. Does it make sense to move the sensors to only create partitions and not trigger jobs? 2. Can we move partition creation to a job? 3. Is there a better way to handle this timeout and resourcing problem?

For context sake we went ahead and 1. Removed having spec_cache as a materialized asset (🤷 ) 2. Increased our schedule to be from every minute to every 5 3. Added queue prioirity tags.

However, After doing all this though @Joe were having a lot of concerns about using Dagster Cloud at Airbyte. For example a schedule definition that 1. Looks for blobs in a gcs folder 2. Gets their etag (without downloading the file) 3. Adds only new etags as dynamic partition keys which to me has a low level of complexity. Runs in 3.8s on my machine (great!) But in Dagster cloud can take anywhere from 48s to well over a minute. (not great 😞 ) Which leads to broken sensors, asset materializations and queue overflow. For what is a task that parses a yaml file into json. I understand that this is the business model, we use a similar one at Airbyte, but I think your resourcing is so low that as a customer 1. I can't confidently develop a basic to intermediate pipeline locally and expect it to work in production 2. Which in turn means I cant then show to my organization and call it a success 3. and the kicker is Im so early in our product cycle that the alternative options to Dagster Cloud are very attainable a. self host (Im already spending time reworking a system that runs on my local machine, why not skip the next rework and use k8s) b. move to another orchestrator (another team is showing promising results with prefect and we havent built such a large Dagster system that porting would be insurmountable) Why am I telling you all this? 1. I like Dagster and want to see you succeed 2. I dont want to tell my org that we made a mistake. 3. This would be the feedback I would hope Airbyte would get before customers churned Im going to forward this to #dagster-feedback as well.