https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
d

Daniel Mosesson

12/05/2022, 6:57 PM
Is there a good page for reference when writing an IO manager? I have written a few that basically take assets and put them into Postgres, and some of the challenges I've run into are: • What is the best way to pass metadata from an
op
to the
io_manager
? What if it needs to be dynamic, e.g. if the op knows what the name of the output should be (setting metadata on the
Output
does not seem to work) • What is the best way to have an
io_manager
that only really works for assets? (thanks to @owen for the idea of the custom decorator) • What are all of the possible sources of the "name" of the thing: ◦
context.asset_key
? ◦
context.step_key
? ◦ various
names
? • detecting partition type (currently I use
instanceof(context.asset_partitions_def
a bunch, maybe there is a better way) • better understanding how
context.asset_partitions_time_window
• How does all of this work in the context of: ◦ Backfills ◦
@multiasset
SourceAsset
DynamicOutput
• Other things to be aware of that may contain sharp edges for those who are writing
io_manager
s Currently what I do is look at the
dagster
test suite, which is pretty awesome, but sometimes doesn't give the comprehensive overview that I sometimes lack. Does something like this already exist/can it?
c

chris

12/09/2022, 6:40 PM
hey! We don't really have any content like this right now - but it is on the radar / planned at some point. cc @jamie who might have more details, or at the very least might find the challenges presented here to be good material
j

jamie

12/09/2022, 7:03 PM
hey @Daniel Mosesson - i dont have much to add at this point. writing a guide like this is on my to do list as part of some other IO manager changes I have in the works. I’ve copied over all of the points you brought up so i can address them in the future guide. If you do have any specific questions as you write io managers, i’m happy to help out or point you to examples in our own IO managers