19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi folks, could someone please guide me how to set...
# announcementss
Sachit Shivam
09/15/2020, 11:11 AMHi folks, could someone please guide me how to set Dagster up so that dagit / dagster, RabbitMQ and Celery workers are all running remotely (i.e.: on separate EC2 instances)?
I have dagit / dagster running on one instance, RabbitMQ on one and multiple instances for Celery workers (need to be able to add more based on workload scaling)
I can't seem to figure out how to provide the correct settings / configuration to dagster
Also, will some special setup be required for Celery? What app name will have to be passed to it via the CLI?
m
max
09/15/2020, 1:38 PMhi @Sachit Shivam -- what issues are you encountering?
s
Sachit Shivam
09/15/2020, 1:39 PMHi @max
I need to set RabbitMQ and Celery workers up on separate instances, and I'd like to know how to modify the dagster.yaml for that
Also, will I have to deploy copies of my pipeline code where I'm running my Celery workers?
m
max
09/15/2020, 1:40 PMyes, your celery workers will need copies of the pipeline code --
dagster-celerymax
09/15/2020, 1:40 PMyour dagster.yaml will need the address of the broker
max
09/15/2020, 1:40 PMand that same dagster.yaml will need to be present on the workers, so they know how to contact the broker and where to stream events to
s
Sachit Shivam
09/15/2020, 1:41 PMOh okay
Alright
Thank you
Can I ping you again if I run into an issue?
m
max
09/15/2020, 1:50 PMyes of course! happy to debug with you
s
Sachit Shivam
09/15/2020, 1:50 PMThank you!
😄
Sachit Shivam
09/16/2020, 6:44 AMHi @max!
I'm trying to start a worker by running
dagster-celery worker start -A dagster_celery.app -y /opt/dagster/app/dagster.yaml Copy code
execution:
celery:
config:
broker: 'pyamqp://guest@ip-172-31-7-185.ap-south-1.compute.internal//'
backend: 'rpc://ip-172-31-7-185.ap-south-1.compute.internal'Sachit Shivam
09/16/2020, 6:45 AMI also tried to format the dagster.yaml as in the documentation:
Copy code
execution:
celery:
config:
broker: '<pyamqp://guest@ip-172-31-7-185.ap-south-1.compute.internal//>'
backend: '<rpc://ip-172-31-7-185.ap-south-1.compute.internal>'35 Views