The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

I consider implementing a custom daemon with complex scheduling logic and dependency management, similar to `AssetDaemon`.

In theory, it is possible to implement this logic using tons of sensors running on every tick. But I am not sure about the performance if I have a few thousand assets to manage eventually.

Is it possible (from architecture standpoint) to add and maintain a custom daemon without changing Dagster core? Did anyone else try to do it? Any examples?

Thank you!

Hmmm! It looks like it is possible to define one super-sensor which can start any various jobs in one repository.

Feature is marked as experimental, but seems to be working.

In theory, it solves my problem. No need for a separate daemon, just one sophisticated sensor is enough.

Hi Vitaly - Let us know if you have any further questions on this!