Hi I m trying to write a custom `io manager` that saves the dagster #ask-community

Hi, I’m trying to write a custom `io_manager` tha...

Andras Somi

01/22/2023, 2:19 PM

Hi, I’m trying to write a custom

io_manager

that saves the output of a partitioned asset and behaves nicely if a non-partitioned downstream asset asks for the output (it should return the latest available partition), but I got a bit lost on what to look for in the

InputContext

object. I tried to decipher the relevant methods on

UPathIOManager

but couldn’t really grasp what’s going on with the partition keys (comments are confusing for me, I even sense some typos there, but cannot be sure…). How should I decide whether the downstream asset is partitioned so

load_input

should return the specific partition or non-partitioned therefore

load_input

should return the latest available?

Andras Somi

01/22/2023, 2:38 PM

Hm, probably I chased myself down to a rabbit hole with

UPathIOManager

. I ended up having something like this in

load_input

Copy code

if context.has_partition_key:
    # Asking for a specific partition
    date_str = context.partition_key
    # ... retrieve output for this key
else:
    # Asking for latest partition
    # ... check all available partitions and retrieve latest

Is this how it should be done?

yuhan

01/28/2023, 1:49 AM

Hey Andras, sorry for the delay. Yes I’d recommend the same. Having the condition check in load_input looks reasonable.

Andras Somi

01/28/2023, 7:19 AM

Great, thanks for confirming!

2 Views

Open in Slack

Previous Next