19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Can anyone point me to documentation about where t...
# ask-communityl
Lawrence Jenkins
04/06/2023, 9:15 PMCan anyone point me to documentation about where to find the source of a daemon_id? I can't seem to find it anywhere
d
daniel
04/07/2023, 4:26 PMHi Lawrence - i'm not actually sure what you mean by daemon_id. Can you give some more information on what you're hoping to do / what problem you're seeing?
l
Lawrence Jenkins
04/07/2023, 4:40 PMWhen I run dagster I get:
dagster.daemon.BackfillDaemon - Error - Another BACKFILL daemon is still sending heartbeats. You likely have multiple daemon processes running at once, which is not supported. Last heartbeat daemon id: 06fd89-1ce6-458b-8d38-7b37c7504e9, Current daemon_id: ef47e9da-d690-4d84-bd0d-15173a9f0a4ad
daniel
04/07/2023, 4:50 PMHow are you running your daemon?
daniel
04/07/2023, 4:51 PMdagster doesn't really have a way of knowing where a daemon is coming from, but we can give some tips how to find it and kill it if we know how it was launched originally
l
Lawrence Jenkins
04/07/2023, 4:52 PMRunning dagster in a docker container
d
daniel
04/07/2023, 4:52 PMIf you run 'docker ps -a" it will show you all running containers
daniel
04/07/2023, 4:53 PMthe daemon will probably be there (assuming it has its own container?)
l
Lawrence Jenkins
04/07/2023, 4:54 PMIt only shows one active container
d
daniel
04/07/2023, 4:55 PMAre you running dagit / the daemon / etc. all in the same container? or does each get their own container?
l
Lawrence Jenkins
04/07/2023, 4:55 PMRunning dagit and the daemon in the same container
d
daniel
04/07/2023, 4:55 PMCan you share the entrypoint / Dockerfile for that container?
daniel
04/07/2023, 4:57 PMoh one other thing - does that "Another BACKFILL daemon is still sending heartbeat" repeat indefinitely? or does it only fire briefly once on startup? if the daemon just went down and was restarted, it's possible that it's seeing the previous daemon that just went down, in which case the error would be harmless
l
Lawrence Jenkins
04/07/2023, 4:57 PMIt repeats indefinitely
Lawrence Jenkins
04/07/2023, 4:58 PMThe "ENTRYPOINT" in the docker file is set to 2 .sh files
d
daniel
04/07/2023, 4:59 PMCan you share them? its possible its set up in such a way that the daemon can somehow run twice
l
Lawrence Jenkins
04/07/2023, 4:59 PMThe .sh files?
d
daniel
04/07/2023, 4:59 PMYeah - do you have something that would make the daemon restart if it fails?
l
Lawrence Jenkins
04/07/2023, 5:01 PMRun_retries is a part of the dagster.yaml, not sure if that is what you mean tho
d
daniel
04/07/2023, 5:02 PMThat would refer to jobs - i'm referring to the way in which you are running your dagster-daemon process
daniel
04/07/2023, 5:02 PMwhich would be in the entrypoint for the docker container
daniel
04/07/2023, 5:02 PMIf you're not able to share that - the problem that you'll need to solve is making sure that at most one dagster-daemon process can be running at any given time within that container
daniel
04/07/2023, 5:04 PM(that's assuming that the only possible place the daemon could be running is in that container - if you spun up a daemon locally in the past somehow pointed at the same storage, that would be something else to rule out)
l
Lawrence Jenkins
04/07/2023, 5:05 PMI don't belive that is the issue, I've only ever ran dagster through docker on my computer
Lawrence Jenkins
04/07/2023, 5:06 PMI can't share the dockerfile unfortunately. Is there any example Dockerfile you could point me to?
Lawrence Jenkins
04/07/2023, 5:07 PMAlso to run dagster in docker properly do you have to use 'docker run coordinator' or can you use queued run coordinator inside a docker container?
d
daniel
04/07/2023, 5:07 PMTypically we recommend running the daemon in its own container like in the example here: https://github.com/dagster-io/dagster/tree/master/examples/deploy_docker
If you made 'dagster dev' your entrypoint, i think it would handle spinning up both dagit and the daemon for you as subprocesses
daniel
04/07/2023, 5:07 PMIt's fine to use the queued run coordinator inside a docker container
👍 1
l
Lawrence Jenkins
04/07/2023, 5:08 PMGotcha, so run dagit and dagster daemon in different contianers?
d
daniel
04/07/2023, 5:08 PMthat can be more complicated but has some nice properties - generally docker recommends running a single process per container if possible
daniel
04/07/2023, 5:09 PMhttps://docs.docker.com/config/containers/multi-service_container/ has some guidelines from them
l
Lawrence Jenkins
04/07/2023, 5:10 PMSo daemon in its own container and the user code in a separate container?
d
daniel
04/07/2023, 5:11 PMthat also has some nice properties - you can update your code without needing to restart everything
l
Lawrence Jenkins
04/07/2023, 5:13 PMI don't think I'm understanding correctly. So it's recommended to run daemon, dagit, and user code all in separate containers?
d
daniel
04/07/2023, 5:13 PMThat's what a lot of production dagster deployments do, yeah
daniel
04/07/2023, 5:13 PMMore information here: https://docs.dagster.io/deployment/guides/docker#multi-container-docker-deployment
l
Lawrence Jenkins
04/07/2023, 5:14 PMGotcha. I'll check it out thank you for the help!
Lawrence Jenkins
04/07/2023, 5:18 PMOne more question sorry. I see that there are 5 daemons running in dagit. When you said "at most one dagster-daemon process" do you mean only 1 of the 5 in dagit be running at a time? Or there should only be 1 of each type running at a time?
d
daniel
04/07/2023, 5:23 PMone daemon process, 5 daemons
daniel
04/07/2023, 5:23 PM(1 of each type)
l
Lawrence Jenkins
04/07/2023, 5:24 PMGotcha, thanks
65 Views