I'm writing an iomanager, is there a good guide to how to deal w/ parallelism/multiprocessing here? Specifically, for time based partitions my iowriter is not parallelizable but for static partitions it is. Ideally, I'd be able to specify this in a way that also works w/ MultiPartitions.

hey @Drew You are you looking for advice for how to determine if an output is from a time partition vs a static partition vs a multi partition?

So, I know that. I'm more concerned w/ how dagster deals w/ multiprocessing. Is there a way to guarantee that a backfill will only be handled by a single thread?

cool, just wanted to make sure i knew which part of the question you were wondering about. You could change the executor you use for the specific assets/jobs that use that io manager to be the inprocess executor instead of the multi-process executor https://docs.dagster.io/deployment/executors#specifying-executors https://docs.dagster.io/deployment/executors#example-executors

Hmmm I think I'm missing something here.

ok i think i understand the problem better now. I agree that changing the executor for every single pipeline doesn’t make sense. We don’t have a built in way to achieve the kind of thing you want. So you could look at other solutions to ensure that two instances of the io manager aren’t writing to the file at the same time. one could be maintaining a lock file that the io manager needs to hold in order to write to the parquet file