Hi All, I started using Dagster, and a scenario came up that I wasn't able to get working; documentation is also absent on that.
We have a schedule that runs periodically. One of the downstream jobs, JobA, from the schedule has an output. We want to persist this output to BigQuery, after some transformations, and thought using an Asset to encapsulate the table made sense. We thought we should be able to connect the output of JobA to the input of the Asset (JobA -> Asset), but it appears the documentation has nothing on this. Hacking at it also hasn't worked so far. Is this a supported case?
03/01/2023, 3:17 AM
I think one way to model this would be to turn JobA into a graph-backed asset, then have your BigQuery asset define the transformations that have to be applied to JobA's output. I don't think assets can depend on jobs directly.