I was wondering if this sort of a set up is possible with `d dagster #announcements

I was wondering if this sort of a set up is possib...

Zuhair Ikram

03/05/2021, 1:49 AM

I was wondering if this sort of a set up is possible with

dagster

? The other two boxes represent hosts with like an agent or a “lite” version of dagster. The agent kicks off processes on the host when main dagster scheduler tells it to.

Noah K

03/05/2021, 4:39 AM

As in you want to implement solids in both arbitrary python and dbt?

Zuhair Ikram

03/05/2021, 6:00 AM

Yes

Noah K

03/05/2021, 6:17 AM

Sure, solids can be written however you need 🙂

Zuhair Ikram

03/05/2021, 6:34 AM

Sorry, let me rephrase. Would I basically do a

pip install dagster

on all three boxes +

pip install dagit

on one box to get this going?

Zuhair Ikram

03/05/2021, 6:35 AM

Is there a write up on how one would set up what I have diagrammed there?

Noah K

03/05/2021, 6:37 AM

running things on multiple servers is unrelated to the language you are writing your solids in

Noah K

03/05/2021, 6:38 AM

Look at things like dagster_celery and dagster_k8s for multi-node execution options at varying levels of complexity 🙂

Zuhair Ikram

03/05/2021, 8:07 AM

Yeah, I think we went down the wrong path for my question. It wasn’t about the language for solids, but more about a multi-node installation. I was trying to compare the complexity for implementing a multi-node installation for dagster vs some alternatives like Airflow2, etc.

Noah K

03/05/2021, 8:10 AM

Overall fairly similar

Noah K

03/05/2021, 8:10 AM

Celery on Kubernetes is the usual way forward 🙂

Noah K

03/05/2021, 8:10 AM

You can mix-n-match it a bit from there

Zuhair Ikram

03/05/2021, 8:16 AM

Thanks!

Noah K

03/05/2021, 8:17 AM

(Dask is also supported but I hateses it and will literally never recommend it 🙂

Open in Slack

Previous Next