I am trying to use Dagster as an orchestrator for ML pipelines. Those pipelines are triggered from a backend when a user requests it via the Python GRPC client, none of the pipelines will be triggered using a schedule. Do I need the Dagster daemon given my use case 🤔?
03/11/2022, 3:30 PM
Hi Javier - you only need the daemon right now if you're using schedules, sensors, or run queueing features that let you control how many runs are being executed at once. We may add other optional features in the future that make use of the daemon though (for example, automatic retries when a pod crashes). If you use our built-in helm chart, it will set up a daemon pod for you automatically.
Javier Llorente Mañas
03/14/2022, 9:20 AM
Ok so in my use case I could erase the Daemon as jobs are going to be trigger via the python grpc client?
moreover, do you have a timeline guys when the python GRPC client will not be an experimental feature anymore?
03/14/2022, 11:10 AM
Do you mean the graphql client? We don't have a specific timeline to share there as it's not a current focus for us, but I don't think it will go away soon or anything, it just may be a while before new APIs are added. It doesn't sound like you need the daemon currently for your use case, but you might miss out on new useful features in the future that use the daemon