Hi All, I'm a newbie in dagster and we are trying to do a PoC to see if dagster is right fit for us.
We have spark (scala) and hive jobs running on a internal thousand node big data cluster. The way we are planning to deploy dagster is to have the dagit installed on the internal cloud environment using kubernetes and docker and have the dagster as service running on the edge nodes of the cluster. Dagit and dagster should then communicate via network where we would need to open some ports between them.
Question -
1. Is this the right approach we are moving forward with?
2. What ports should be opened between the internal cloud and bigdata cluster?