any tips and tricks how to use dagster for tasks with different resources specifications, without "idle" workers but with on demand computation env? we are on aws (aws batch, could go aws eks)

If all you need is to have a single step of a pipeline submit a job to aws batch can have a solid submit it using boto3 and the vcpu/mem requirements are config options for that solid that are passed with the client:

client = boto3.client('batch')
response = client.submit_job(jobDefinition=context.solid_config['job_definition'],
                                 jobName=context.solid_config['job_name'],
                                 jobQueue=context.solid_config['job_queue'],
                                 containerOverrides={'vcpus': solid_config['vcpus'],
                                                     'memory': solid_config['memory']})

From there you can use a botocore waiter to hold the pipeline up until the batch job is completed or use a dagster sensor to trigger another pipeline to start when the output appears on s3

thanks a lot for your ideas!

I guess it really depends on how heavy of a workload each solid has to deal with. Ive been working with dagster directly doing the steps with the lighter loads/bookeeping and submitting the heavier steps to batch. But I've been keeping any additional scripts that the batch job needs contained in the docker image that is used for the job and the "fetch and run" script just wraps the script with the location of the inputs. Was a lot easier to update the docker image than constantly push updates to s3