Hello I m just learning Dagster I have a pipeline that runs

Question

Hello I m just learning Dagster I have a pipeline that runs a script that sends log messages to stdout When I run it without celery I see these log messages in the Dagit run stdout for the solid But when I run it using celery the messages print out to the window where I m running the celery worker and Dagit doesn t see it How can I preserve and access these messages when running my pipeline using celery

claire · Answer

Hi Gillian this guide might be helpful to you <https docs dagster io deployment guides celery execute on celery> If you run the celery worker in the same directory as you saved the pipeline file celery logs should populate in Dagit

Dan Stoner · Answer

< claire> I have reproduced the problem < Gillian Sharer> reported with celery executor and the pipeline declaring `celery mode defs` as indicated in that documentation I get a directory full of `0` byte files

Dan Stoner · Answer

instead of actual logs

Dan Stoner · Answer

```ls latr dagster home storage c2345b9e be27 420c 88d1 ab0a5a17bef1 compute logs total 0 rw r r 1 dstoner staff 0 Oct 5 19 12 get dates out rw r r 1 dstoner staff 0 Oct 5 19 12 create archive cache out rw r r 1 dstoner staff 0 Oct 5 19 12 create mustang cache out rw r r 1 dstoner staff 0 Oct 5 19 12 get dates err rw r r 1 dstoner staff 0 Oct 5 19 12 create archive cache err rw r r 1 dstoner staff 0 Oct 5 19 12 create mustang cache err rw r r 1 dstoner staff 0 Oct 5 19 12 get station list out rw r r 1 dstoner staff 0 Oct 5 19 12 get station list err rw r r 1 dstoner staff 0 Oct 5 19 12 create archive cache complete rw r r 1 dstoner staff 0 Oct 5 19 12 get dates complete rw r r 1 dstoner staff 0 Oct 5 19 12 create mustang cache complete rw r r 1 dstoner staff 0 Oct 5 19 12 get station list complete rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II ABPO out rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II ALE out rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II ABPO err rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II ALE err rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II ARTI out rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II ARTI err rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II ASCN out rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II ASCN err rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II BORK out rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II BORK err rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II CMLA out rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II BORG out rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II CMLA err rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II BFO out rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II BORG err rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II BFO err rw r r 1 dstoner staff 0 Oct 5 19 12 run metrics for one II ASCN complete ```

Dan Stoner · Answer

No output in dagit

johann · Answer

Hi Dan have you configured a ComputeLogManager on your instance

Dan Stoner · Answer

Have not touched that since the default is supposed to be local files or some such thing

Dan Stoner · Answer

I still don t have a good handle on the terminology and boundaries for all these configurations repository vs workspace vs instance exploding head

Dan Stoner · Answer

If there is a diagram somewhere would appreciate seeing it

johann · Answer

We hear you on the configuration complexity Definitely something that s on our mind and we appreciate any feedback

johann · Answer

Let me jump in to the specific issue first then we could circle back to the overall system question For this issue are the celery workers running on the same host as Dagit As in they have access to the same filesystem

johann · Answer

Configuring compute log storage is the solution when this isn t the case which is why I brought that up But it seems like you re on one box <https docs dagster io deployment dagster instance compute log storage>

Dan Stoner · Answer

yes on the same system our development laptops

johann · Answer

Got it

johann · Answer

And just to be concrete this is with the DefaultRunLauncher When using inprocess or multiprocess executors the logs show up When using the celery executor they don t

johann · Answer

And the Mode you re using for celery has the same logging configuration

Dan Stoner · Answer

This is where having config scattered everywhere makes answering your questions a challenge

Dan Stoner · Answer

gt When using inprocess or multiprocess executors the logs show up YES

Dan Stoner · Answer

gt When using the celery executor they don t YES

Dan Stoner · Answer

I was going to say we switched to the Queued Run Launcher in order to use celery but I m looking and that doesn t seem to be a thing

Dan Stoner · Answer

Pretty sure if I didn t need to change from defaults I didn t

Dan Stoner · Answer

Ah We had to use `QueuedRunCoordinator` in order to use Celery

Dan Stoner · Answer

I searched my code repo for RunLauncher and didn t find anything so I assume we are using the default

johann · Answer

thumbsup

johann · Answer

gt Ah We had to use `QueuedRunCoordinator` in order to use Celery I don t believe this is a requirement I m curious what you ran in to

Dan Stoner · Answer

Oh maybe I needed concurrency limits to prevent Celery from destroying my machine

johann · Answer

Got it

Dan Stoner · Answer

but I sorta feel like no tasks get submitted to the broker without changing the Run Coordinator

johann · Answer

When you ran pipeline without celery did you have a ModeDefinition defined Or was it just defaults

Dan Stoner · Answer

when not actually using celery executor it works whether ModeDefinition is defined or not 1 Initially we didn t have ModeDefinition at all no celery could see logs 2 added ModeDefinition for celery could not see logs 3 toggled the executor to multiprocess but didn t change the pipeline def so it still has ModeDefinition and can still see the logs

johann · Answer

That s helpful thanks

Dan Stoner · Answer

Having to paste config into the playground is annoying I wish I knew where to actually put some of this kind of config ```execution multiprocess config max concurrent 3```

Dan Stoner · Answer

since to test I want to toggle that back to the Celery executor but that isn t stored anywhere in config management

Dan Stoner · Answer

Not sure what part of the system created the ` out` files for those jobs the `0` byte ones I mentioned Celery or the job manager

johann · Answer

gt Having to paste config into the playground is annoying Currently those can be stored in code in PresetDefinitions under run config We re very aware that there are a bunch of nouns and concepts here and in a few weeks we ll be releasing a revamp that seeks to simplify things <https docs dagster io apidocs experimental>

johann · Answer

gt Not sure what part of the system created the ` out` files for those jobs the `0` byte ones I mentioned Celery or the job manager That would be Dagster s compute log manager

johann · Answer

< prha> could I put this on your radar to try to repro Logs not showing up with celery executor and local compute logs

prha · Answer

< Dagster Bot> issue Compute logs empty for runs executed with the celery executor

Dagster Bot · Answer

Created issue at <https github com dagster io dagster issues 5048>

Dan Stoner · Answer

One more thing I believe most all of the job output is STDOUT from a Docker container ```subprocess run docker run ```

Dan Stoner · Answer

^^^ I will add this to the github issue

Dan Stoner · Answer

also nice Slack bot

Dan Stoner · Answer

It appears that the only thing needed to break capture of Celery logs is this import statement ```import subprocess``` I added this to the github issue

johann · Answer

Wow bizarre

johann · Answer

But this import doesn t break it with inprocess multiprocess executor

Dan Stoner · Answer

I will verify It also seems that any time the code changes the celery worker needs to be restarted

johann · Answer

That s correct since the celery workers run the code in process Ideally celery wouldn t be part of your dev flow so you could avoid that but it would also be possible to setup hot reload that restarts the workers for you

Dan Stoner · Answer

I might be retracting my bizarre statement

Dan Stoner · Answer

I now think the most specific description is that output from print statements aka STDOUT from a job executed in celery is lost not captured Period

Dan Stoner · Answer

Confirmed when running exactly the same in Playground except one change of removing execution celery config I get the expected outputs in a captured log