Expoze Dummy

03/01/2023, 4:33 PM
Hi, dagster team! Is there a way to possibly connect an asset with an op? I have generated an asset that reads a csv file from s3 and I want to use greatexpectations (dagster-ge) op to test this one out. Thanks 🙂
:dagster-bot-resolve: 1


03/01/2023, 7:12 PM
One idea would be to model the
op as a graph-backed asset, with the results of the QC checks being your asset. Then the GreatExpectations asset graph could depend on the asset you want to QC. Another idea (might be a little more of a hack) that I personally haven't tried before would be to use the new `root_input_manager` abstraction to load the asset by just embedding the same logic the upstream asset's IOManager uses to load the asset's value. This wouldn't be recognized as a dependency between the op and the asset by Dagster, which may or may not be what you want (the graph-backed asset approach would show your
asset as stale every time you materialized the upstream, which may or may not be what you want). Other than that I don't think there's a Dagster-native way to load an asset from an op, but I think it's a feature that's on the Dagster team's radar as it comes up pretty regularly.


03/01/2023, 10:11 PM
A variant on Zach's second suggestion that might be a little easier is this capability that we recently added:
❤️ 1

Expoze Dummy

03/03/2023, 4:19 PM
Hey Zach, Sandy, thank you so much for your inputs. After quite a lot of internalization, I just converted the dagster-ge op into an asset using
. I'll see if this will work 🙂

Terry Dontje

04/17/2023, 2:47 PM
I am trying something similar, but I did try and incorporate a ge op made with get_validation_op_factory in a graph_asset as Sandy suggested. However when the op is executed it looks like it is erroring out because resource_defs does not have ge_data_context defined. Which is true because I haven't figured out how to pass resource_defs into the asset job definition which calls the graph_asset. Am I missing something key to get the ge_op to work?