hello all i am working on dagster celery executor i am able

Question

hello all i am working on dagster celery executor i am able to run pipeline by celery executor of dagster version==0 13 but same process not working for me of dagster version==0 12 14 please let me know anyone have any idea i am following this docs gt <https docs dagster io 0 12 14 deployment guides celery>

daniel · Answer

Hi Rahul could you share more details about what didn t work reproduction steps the nature of the failure and the stack traces of any error messages you saw

Rahul Sharma · Answer

i saw there is no error i think dagster unable to communicate with celery executor same process is working for me of dagster V==13 12

Rahul Sharma · Answer

these all screenshot related dagster v=13 12

Rahul Sharma · Answer

but when i change the dagster 13 12 to 12 14 then there is nothing on celery terminal and no data showing on flower please do help me

Rahul Sharma · Answer

```from dagster celery import celery executor from dagster import ModeDefinition default executors fs io manager pipeline solid celery mode defs = ModeDefinition resource defs= io manager fs io manager executor defs=default executors + celery executor solid def get name return dagster solid def get name one return dagster solid def get name two return dagster solid def get name three return dagster pipeline mode defs= ModeDefinition executor defs=default executors + celery executor pipeline mode defs=celery mode defs def parallel job get name get name one get name three get name two ```

Rahul Sharma · Answer

< daniel> please have a look above

daniel · Answer

It might take a little longer than normal to respond to your question due to the holidays but when I m back at a keyboard I can try to reproduce on 0 12 14 I assume there s some reason you are stuck on on 0 12 14 I don t think there are breaking changes or any migrations required to switch to 0 13

Rahul Sharma · Answer

ok thanks for your response

daniel · Answer

ok from your screenshots it looks like your job is still finishing it s just not using the celery executor What are you using as your run config in dagit

daniel · Answer

you need to set the following config in dagit to tell it to use celery ```execution celery ``` that would be true in 0 13 12 too though are you sure you used the same run config and job code in the two different dagster versions

Rahul Sharma · Answer

```execution celery broker <pyamqp admin pwd123 0 0 0 0 5672 > backend rpc include my module ```

Rahul Sharma · Answer

this is the file i am using but i am not using any custom config

daniel · Answer

for dagit to know that it should use the celery executor you need to specify run config

daniel · Answer

in both 0 12 14 and 0 13 12 if you re using the pipeline decorator

daniel · Answer

so you will want to make sure that in dagit you use that config you posted when you launch the run ```execution celery broker <pyamqp admin pwd123 0 0 0 0 5672 > backend rpc include my module ```

Rahul Sharma · Answer

```dagster celery worker start y celery config yaml```

Rahul Sharma · Answer

i used above command but not working in v=0 12 14 but everything is working in v=0 13 12

Rahul Sharma · Answer

do my module py will be same for both version

Rahul Sharma · Answer

can we connect for while i think it would be better

daniel · Answer

can you answer my question above about what run config you set on the run in dagit Can you paste what you typed into the Launchpad box in the dagit UI The run config in dagit needs to include the same execution config that you specified when starting the celery workers

Rahul Sharma · Answer

```execution celery broker <pyamqp admin pwd123 0 0 0 0 5672 > backend rpc include my module ```

Rahul Sharma · Answer

this one

Rahul Sharma · Answer

``` $ dagster celery worker start A <http dagster celery app|dagster celery app> $ docker run p 5672 5672 rabbitmq 3 8 2 $ dagit f celery pipeline py $ celery flower A dagster celery app port=5555```

Rahul Sharma · Answer

i am using these command working on v= 13 but not v= 12

daniel · Answer

when you launch a run in dagit there is a box on the Launchpad where you enter configuration for the run here is a screenshot of it

daniel · Answer

in that box you need to tell it to launch the run in celery by pasting the run configuration there ```execution celery broker <pyamqp admin pwd123 0 0 0 0 5672 > backend rpc include my module ```

daniel · Answer

<https docs dagster io concepts configuration config schema dagit> says more about how to apply configuration to runs

Rahul Sharma · Answer

but why it is working on v= 13 whitout doing anything

daniel · Answer

did you change the code at all between v13 and v12

daniel · Answer

the code of your dagster job pipeline i mean

Rahul Sharma · Answer

i am using below two codes for v=0 13 and v=0 12

daniel · Answer

Got it it s because job and pipeline handle defaults differently The confusing situation you ran into in v12 has been fixed in v13

daniel · Answer

and less config is required to make it run using celery by default

daniel · Answer

so if using job and v13 is an option for you it has fixed a number of problems and confusing things including this one

Rahul Sharma · Answer

so what should i do now

daniel · Answer

I gave you some steps above for how to launch your run in dagit using celery in 0 12 14 paste your config into the box I posted a screenshot of then press launch run

daniel · Answer

or switch to 0 13 this is my top recommendation if that is an option

Rahul Sharma · Answer

got it thanks

Rahul Sharma · Answer

is there any way directly load this config file from pipeline script

daniel · Answer

You can use this API to configure the celery executor in code and the code can load the config from a file <https docs dagster io concepts configuration configured|https docs dagster io concepts configuration configured>