https://dagster.io/ logo
m

Miri Weissler

12/31/2020, 7:10 AM
Hi, Does integration with DataFlow (apache beam) on the roadmap?
s

sandy

12/31/2020, 7:11 PM
Hi Miri - it's currently possible to run DataFlow code inside Dagster solids. Are there particular capabilities you'd find useful inside an integration?
g

gatesma

01/11/2021, 5:52 PM
Could Dagster work with a Java dataflow also .?
s

sandy

01/11/2021, 8:48 PM
One approach for using dagster with Java dataflow would be to use the dagster-shell package to have dagster invoke shell commands that launch Java dataflow jobs. This is essentially what the dagster-spark package does to launch Scala Spark jobs. https://docs.dagster.io/_apidocs/libraries/dagster_shell#shell-dagster-shell
5 Views