https://dagster.io/ logo
#announcements
Title
# announcements
j

J Groß

03/18/2020, 12:16 PM
Hi everyone, I'm experimenting and learning about dagster for a few days now. So far I was just using it to run dags on my local machine, which was an exciting experience. Currently I'm wondering how I would integrate a new scheduler and execution engine. Say I have a running htcondor cluster and want to use it for sceduling and execution (it already supports dags on its own). Is there further documentation you could kindly point me out to? How do I start? The documentation mentions to say Hi and ask here 🙂
a

Auster Cid

03/18/2020, 2:08 PM
Hi @J Groß, Im also a beginner to dagster, but from what I've read here these last weeks, you'll be wanting to look into implementing your own implementation of a RunLauncher
Not 100% sure though, as I still execute everything locally
a

alex

03/18/2020, 2:48 PM
The info Auster points out above will be the tools you need for “launching” the run over in to the htcondor cluster and how to set up run/event storage for distributed execution. As for mapping dagster DAG execution on to htcondor’s you can look at writing your own engine. The best way to do that at this point is to reference the existing engine integrations in the repo. Celery, our most recent https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-celery/dagster_celery/engine.py airflow/dask https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-airflow https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dask/dagster_dask/engine.py
j

J Groß

03/18/2020, 3:54 PM
Thank you @Auster Cid and @alex! I'll give it a try.