Hi! I'm trying to configure the celery-executor with Amazon SQS but it seems that celery does not have support for a results backend with sqs. Does this mean the backend is not necessary? could someone point me to a configuration that might work with dagster-celery and sqs?
06/27/2020, 2:38 PM
you will need to use a system other than SQS for the results backend -- i suspect your best bet in this case is a database
06/30/2020, 3:32 AM
Thanks for the reply, so I can use sqs for the broker, and use another thing for the results?
Also, is it possible to specify queues per pipeline? I tried with predetermined queues but this seems to fail. Using SQS, a new queue is created per run.