It would be great to have a way to control concurrency of operators by specifying resource requirement as in luigi.

Yeah that’s a nice api. Something like that is on our radar