https://dagster.io/ logo
#ask-community
Title
# ask-community
l

Lawrence Jenkins

04/06/2023, 9:15 PM
Can anyone point me to documentation about where to find the source of a daemon_id? I can't seem to find it anywhere
d

daniel

04/07/2023, 4:26 PM
Hi Lawrence - i'm not actually sure what you mean by daemon_id. Can you give some more information on what you're hoping to do / what problem you're seeing?
l

Lawrence Jenkins

04/07/2023, 4:40 PM
When I run dagster I get:
dagster.daemon.BackfillDaemon - Error - Another BACKFILL daemon is still sending heartbeats. You likely have multiple daemon processes running at once, which is not supported. Last heartbeat daemon id: 06fd89-1ce6-458b-8d38-7b37c7504e9, Current daemon_id: ef47e9da-d690-4d84-bd0d-15173a9f0a4a
d

daniel

04/07/2023, 4:50 PM
How are you running your daemon?
dagster doesn't really have a way of knowing where a daemon is coming from, but we can give some tips how to find it and kill it if we know how it was launched originally
l

Lawrence Jenkins

04/07/2023, 4:52 PM
Running dagster in a docker container
d

daniel

04/07/2023, 4:52 PM
If you run 'docker ps -a" it will show you all running containers
the daemon will probably be there (assuming it has its own container?)
l

Lawrence Jenkins

04/07/2023, 4:54 PM
It only shows one active container
d

daniel

04/07/2023, 4:55 PM
Are you running dagit / the daemon / etc. all in the same container? or does each get their own container?
l

Lawrence Jenkins

04/07/2023, 4:55 PM
Running dagit and the daemon in the same container
d

daniel

04/07/2023, 4:55 PM
Can you share the entrypoint / Dockerfile for that container?
oh one other thing - does that "Another BACKFILL daemon is still sending heartbeat" repeat indefinitely? or does it only fire briefly once on startup? if the daemon just went down and was restarted, it's possible that it's seeing the previous daemon that just went down, in which case the error would be harmless
l

Lawrence Jenkins

04/07/2023, 4:57 PM
It repeats indefinitely
The "ENTRYPOINT" in the docker file is set to 2 .sh files
d

daniel

04/07/2023, 4:59 PM
Can you share them? its possible its set up in such a way that the daemon can somehow run twice
l

Lawrence Jenkins

04/07/2023, 4:59 PM
The .sh files?
d

daniel

04/07/2023, 4:59 PM
Yeah - do you have something that would make the daemon restart if it fails?
l

Lawrence Jenkins

04/07/2023, 5:01 PM
Run_retries is a part of the dagster.yaml, not sure if that is what you mean tho
d

daniel

04/07/2023, 5:02 PM
That would refer to jobs - i'm referring to the way in which you are running your dagster-daemon process
which would be in the entrypoint for the docker container
If you're not able to share that - the problem that you'll need to solve is making sure that at most one dagster-daemon process can be running at any given time within that container
(that's assuming that the only possible place the daemon could be running is in that container - if you spun up a daemon locally in the past somehow pointed at the same storage, that would be something else to rule out)
l

Lawrence Jenkins

04/07/2023, 5:05 PM
I don't belive that is the issue, I've only ever ran dagster through docker on my computer
I can't share the dockerfile unfortunately. Is there any example Dockerfile you could point me to?
Also to run dagster in docker properly do you have to use 'docker run coordinator' or can you use queued run coordinator inside a docker container?
d

daniel

04/07/2023, 5:07 PM
Typically we recommend running the daemon in its own container like in the example here: https://github.com/dagster-io/dagster/tree/master/examples/deploy_docker If you made 'dagster dev' your entrypoint, i think it would handle spinning up both dagit and the daemon for you as subprocesses
It's fine to use the queued run coordinator inside a docker container
👍 1
l

Lawrence Jenkins

04/07/2023, 5:08 PM
Gotcha, so run dagit and dagster daemon in different contianers?
d

daniel

04/07/2023, 5:08 PM
that can be more complicated but has some nice properties - generally docker recommends running a single process per container if possible
l

Lawrence Jenkins

04/07/2023, 5:10 PM
So daemon in its own container and the user code in a separate container?
d

daniel

04/07/2023, 5:11 PM
that also has some nice properties - you can update your code without needing to restart everything
l

Lawrence Jenkins

04/07/2023, 5:13 PM
I don't think I'm understanding correctly. So it's recommended to run daemon, dagit, and user code all in separate containers?
d

daniel

04/07/2023, 5:13 PM
That's what a lot of production dagster deployments do, yeah
l

Lawrence Jenkins

04/07/2023, 5:14 PM
Gotcha. I'll check it out thank you for the help!
One more question sorry. I see that there are 5 daemons running in dagit. When you said "at most one dagster-daemon process" do you mean only 1 of the 5 in dagit be running at a time? Or there should only be 1 of each type running at a time?
d

daniel

04/07/2023, 5:23 PM
one daemon process, 5 daemons
(1 of each type)
l

Lawrence Jenkins

04/07/2023, 5:24 PM
Gotcha, thanks
41 Views