Hi everyone, I'm experimenting and learning about dagster for a few days now. So far I was just using it to run dags on my local machine, which was an exciting experience. Currently I'm wondering how I would integrate a new scheduler and execution engine. Say I have a running htcondor cluster and want to use it for sceduling and execution (it already supports dags on its own). Is there further documentation you could kindly point me out to? How do I start? The documentation mentions to say Hi and ask here 🙂

Hi @J Groß, Im also a beginner to dagster, but from what I've read here these last weeks, you'll be wanting to look into implementing your own implementation of a RunLauncher

The info Auster points out above will be the tools you need for “launching” the run over in to the htcondor cluster and how to set up run/event storage for distributed execution. As for mapping dagster DAG execution on to htcondor’s you can look at writing your own engine. The best way to do that at this point is to reference the existing engine integrations in the repo. Celery, our most recent https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-celery/dagster_celery/engine.py airflow/dask https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-airflow https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dask/dagster_dask/engine.py

Thank you @Auster Cid and @alex! I'll give it a try.