Is dagster suitable for never ending job (like worker)? One use case is running streaming job by databricks. What is your recommendation?

I’d recommend modeling it as a sensor which looks for new changes every N seconds and then kicks off runs on new changes — We currently don’t have a native support for streaming. So this micro batch pattern might be the best option.

Thank you @yuhan, but what I mean is not livestream data using Dagster. But to run a streaming job on Databricks and use Dagster to “monitor” it and collect log, which has to run endlessly.

When you say monitor, is there an external state that you’d be actively listening to?

Yes. I trigger a Databricks realtime job and then listen to it.