Hello team, I've noticed that the default file system's I/O uses special characters such as "." and "[]" in file names that are used for data storage. Is there a specific rationale behind using these special characters in file names? Also, would it be possible to alter this naming convention?

hi @Jueming Liu! are these special characters showing up when you are storing assets, or regular op outputs? for assets, the filepath should be based off of the asset's unique key + the partition key (if relevant). these keys can contain any characters, and there's no special logic to filter out certain characters. I'm not aware of any specific rationale beyond the fact that this is the most straightforward solution (and different systems differ in terms of which special characters they support / don't support)

Hi @owen, sorry I didn't make it clear, it's not about customised key but splitter of the path. this path could be the input and output path between ops using default I/O manager For example, I've seen following log : Loading file from: /home/ubuntu/dagster_storage/storage/6218ecd1-4e37-42a0-9ae0-6ce3202eacd4/daily_market_feather_path.market_feather_path.split_res[113615]/run_id Basically, "." is used to split graph name and underlying ops. "[]" is used to indicate the "mapping_key" in my case.

ah I see -- the scheme for generating a filename for an op's output is basically just "concatenate the run id with the step key and the output name" (as this is a simple way to get a unique identifier). step keys in dagster can have special characters, and those aren't sanitized in any special way, for reasons mentioned above (basically, many filesystems allow those characters)

got it. thanks for the explanation!

hm one option here would be to subclass the default UPathIOManager and override the linked function to create a different UPath for an op (in this case, you could substitute

[

with

_

)

That's indeed an option. Thanks for the info