I was wondering if this sort of a set up is possible with `d

Question

I was wondering if this sort of a set up is possible with `dagster` The other two boxes represent hosts with like an agent or a lite version of dagster The agent kicks off processes on the host when main dagster scheduler tells it to

Noah K · Answer

As in you want to implement solids in both arbitrary python and dbt

Zuhair Ikram · Answer

Yes

Noah K · Answer

Sure solids can be written however you need slightly smiling face

Zuhair Ikram · Answer

Sorry let me rephrase Would I basically do a `pip install dagster` on all three boxes + `pip install dagit` on one box to get this going

Zuhair Ikram · Answer

Is there a write up on how one would set up what I have diagrammed there

Noah K · Answer

running things on multiple servers is unrelated to the language you are writing your solids in

Noah K · Answer

Look at things like dagster celery and dagster k8s for multi node execution options at varying levels of complexity slightly smiling face

Zuhair Ikram · Answer

Yeah I think we went down the wrong path for my question It wasn t about the language for solids but more about a multi node installation I was trying to compare the complexity for implementing a multi node installation for dagster vs some alternatives like Airflow2 etc

Noah K · Answer

Overall fairly similar

Noah K · Answer

Celery on Kubernetes is the usual way forward slightly smiling face

Noah K · Answer

You can mix n match it a bit from there

Zuhair Ikram · Answer

Thanks

Noah K · Answer

Dask is also supported but I hateses it and will literally never recommend it slightly smiling face