19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Can dagster be used for spark streaming jobs?
We'r...
# announcementsj
Joshua Marango
03/05/2020, 4:33 PMCan dagster be used for spark streaming jobs?
We're currently using dagster for our analytics report pipelines, works like a charm, debugging is so easy and straightforward.
Will be building a bunch of internal data products with it in the near future, so will update you guys
a
alex
03/05/2020, 4:39 PMHmm - how do you imagine using dagster in the context of a streaming job?
alex
03/05/2020, 4:40 PMDagster is quite batch oriented at this point, so its hard for me to imagine how this would work, but I think it depends on how exactly you are using the streaming jobs
alex
03/05/2020, 4:45 PMyou could have dagster pick up the batched chunks that come out of the streaming job and manage the subsequent processing.
j
Joshua Marango
03/05/2020, 4:45 PMIm using the spark structured streaming API - so its essentially a continuous running batch job done in increments
I have a job that pulls data from AWS SQS and then destructures the JSON then loads the data into a delta table/S3
Joshua Marango
03/05/2020, 4:46 PMhmm, how would dagster pick up the batched chunks?
a
alex
03/05/2020, 4:48 PMdepends on how you have dagster deployed -
dagitalex
03/05/2020, 4:48 PManother option would be to have a scheduled job that uses
should_executej
Joshua Marango
03/05/2020, 4:49 PMahh right, so is GraphQL subscription and streaming pipelines on your guys' vision/timetable
a
alex
03/05/2020, 4:51 PMdagitalex
03/05/2020, 4:55 PMas for how we will support very long lived streaming pipelines - nothing in the plans at this time
👍 1
7 Views