Hi, Does integration with DataFlow (apache beam) on the roadmap?
s
sandy
12/31/2020, 7:11 PM
Hi Miri - it's currently possible to run DataFlow code inside Dagster solids. Are there particular capabilities you'd find useful inside an integration?
g
gatesma
01/11/2021, 5:52 PM
Could Dagster work with a Java dataflow also .?
s
sandy
01/11/2021, 8:48 PM
One approach for using dagster with Java dataflow would be to use the dagster-shell package to have dagster invoke shell commands that launch Java dataflow jobs. This is essentially what the dagster-spark package does to launch Scala Spark jobs.
https://docs.dagster.io/_apidocs/libraries/dagster_shell#shell-dagster-shell