John Helewa
06/26/2020, 12:47 AMCris
06/26/2020, 12:53 AMJohn Helewa
06/26/2020, 12:55 AMalex
06/26/2020, 2:40 PMrun_config
of each pipeline run you can configure how data is passed from solid to solid. In the celery
executor you need something remote all the workers can access so you have something like:
executor:
celery: ...
storage:
s3:
To try something lighter you would do
executor:
multiprocess:
storage:
filesystem:
which wont use celery for execution but instead just a new child process for each stepJohn Helewa
06/26/2020, 5:12 PM