Hi folks could someone please guide me how to set Dagster up

Question

Hi folks could someone please guide me how to set Dagster up so that dagit dagster RabbitMQ and Celery workers are all running remotely i e on separate EC2 instances I have dagit dagster running on one instance RabbitMQ on one and multiple instances for Celery workers need to be able to add more based on workload scaling I can t seem to figure out how to provide the correct settings configuration to dagster Also will some special setup be required for Celery What app name will have to be passed to it via the CLI

max · Answer

hi < Sachit Shivam> what issues are you encountering

Sachit Shivam · Answer

Hi < max> I need to set RabbitMQ and Celery workers up on separate instances and I d like to know how to modify the dagster yaml for that Also will I have to deploy copies of my pipeline code where I m running my Celery workers

max · Answer

yes your celery workers will need copies of the pipeline code `dagster celery` provides a wrapper CLI that you can use to run the workers

max · Answer

your dagster yaml will need the address of the broker

max · Answer

and that same dagster yaml will need to be present on the workers so they know how to contact the broker and where to stream events to

Sachit Shivam · Answer

Oh okay Alright Thank you Can I ping you again if I run into an issue

max · Answer

yes of course happy to debug with you

Sachit Shivam · Answer

Thank you smile

Sachit Shivam · Answer

Hi < max> I m trying to start a worker by running `dagster celery worker start A dagster celery app y opt dagster app dagster yaml` Here is the config ```execution celery config broker pyamqp guest ip 172 31 7 185 ap south 1 compute internal backend rpc ip 172 31 7 185 ap south 1 compute internal ``` However the worker is trying to connect to ` 127 0 0 1 5672 ` instead

Sachit Shivam · Answer

I also tried to format the dagster yaml as in the documentation ```execution celery config broker backend ``` but then it throws this error instead