https://dagster.io/ logo
#ask-community
Title
# ask-community
t

Tobias Macey

02/14/2024, 2:08 PM
I'm in a tricky position of trying to bridge the divide between assets and ops in a pipeline that I'm building. I've got an externally materialized asset (a periodic file drop in GCS) that I wanted to treat as a partitioned asset via a sensor. The downstream operation needs to unpack the downloaded file and generate a downstream set of assets that are dynamically partitioned based on what is included in the archive. Unfortunately there isn't yet a way to treat the downstream outputs as assets, so I have to write it as an op with in-line asset materializations. The tricky piece is that I'm trying to use an IO Manager to load the external asset into the op, but I can't get a handle of the asset key in the IO Manager. All of the relevant code is in my PR here https://github.com/mitodl/ol-data-platform/pull/991/files
For now I hard-coded the asset key in the I/O manager, but that's obviously not ideal.
Has anyone else had any luck with the external assets functionality and having to treat with those assets via an op?
@sean it seems like you're working pretty closely in this area if you happen to have any thoughts on where I'm going wrong.
Or @sandy
s

sandy

02/15/2024, 12:30 AM
Hey Tobias - one thing that might be helpful here is the ability to have an op-based job include an input that comes from an asset: https://docs.dagster.io/guides/dagster/how-assets-relate-to-ops-and-graphs#op-graphs-that-read-from-an-asset Another option would be to pass config to your op-based job that specifies which asset it depends on
t

Tobias Macey

02/15/2024, 1:49 PM
Would that help with populating the asset key in the
InputContext
for the
InputManager
?
s

sandy

02/15/2024, 3:34 PM
Yes, it should