is it possible to fan out multiple times? for instance, we're chunking a list of keys out to 60 docker instances to run. We then want to run a for loop over the batch of keys and run additional ops. Could I potentially call that like this:

result = (
        chunking_op(recruiter_teams_to_score)
        .map(
             lambda batch: append_scores(
                    (
                        keys_in_batch(batch)
                        .map(
                            lambda key: score_batch(
                                scored_data_for_key(key),
                                trained_model,
                            )
                        )
                        .collect()
                    )
                )
        )
        .collect()
    )
    return append_scores(result)

Hi @Danny Steffy - alas, this isn't currently possible. Here's where we're tracking it: https://github.com/dagster-io/dagster/issues/4364 Would it be possible to flatten these into a single level of keys?

hm, like instead of chunking it into batches initially, just run it on each individual key?

right - because you'll end up with 30k steps either way, right?

Right. Is there a way to specify certain steps should be processed with a different run launcher? I don't want every step to be in it's own docker container, I think that would be too much overhead

Is there a way to specify certain steps should be processed with a different run launcher? I don't want every step to be in it's own docker container, I think that would be too much overhead

currently, all steps within a run are launched in the same way