19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
I'm using Google Cloud to run dagster daemon and d...
# announcementsl
Laura Moraes
03/03/2021, 10:25 PMI'm using Google Cloud to run dagster daemon and dagit. For the moment, I'm running it inside a tmux to analyze how everything is running. In my pipeline, we're getting some JSON data and posting it to Big Query every minute. Usually, the whole process takes 4s. However, analyzing my CPU utilization on Google Monitoring it keeps increasing in a constant rate. What can we do to keep CPU down? My impression is that dagster is open more process without closing old ones or something like this.
d
daniel
03/03/2021, 10:26 PMHi Laura - a couple of follow-up questions about this - which version of dagster are you on, and are you using any schedules and sensors currently?
daniel
03/03/2021, 10:45 PM(Depending on your version, there are a few recent improvements we've made that could help with this)
j
João Luiz Carabetta
03/04/2021, 2:43 PMHi @daniel, I work with Laura.
João Luiz Carabetta
03/04/2021, 2:43 PMWe are using dagster 0.10.7, which is the latest one, right?
João Luiz Carabetta
03/04/2021, 2:44 PMThere is a cron scheduler (* * * * *) calling the job
João Luiz Carabetta
03/04/2021, 2:46 PMDagster is running for 17h but there are 1082 threads opened
João Luiz Carabetta
03/04/2021, 2:46 PMfull htop pic
d
daniel
03/04/2021, 2:55 PMGot it - are you using any sensors? Or just the scheduler?
j
João Luiz Carabetta
03/04/2021, 3:02 PMjust the scheduler
d
daniel
03/04/2021, 3:10 PMGot it - I'll do some investigation and see if I can reproduce this. We've actually made some big improvements in our process management in the daemon in 0.10.8 (going out later today) but I need to confirm whether this issue will be fixed as part of that.
daniel
03/04/2021, 3:10 PMWould it be possible to post your dagster.yaml for repoduction purposes?
j
João Luiz Carabetta
03/04/2021, 3:11 PMd
daniel
03/04/2021, 3:12 PMah perfect, thanks! Will report back.
j
João Luiz Carabetta
03/04/2021, 3:18 PMwe are still figuring out dagster, so code structure might not be the best
João Luiz Carabetta
03/04/2021, 3:19 PMwe also had another problem with broken imports, but we can discuss that later
d
daniel
03/04/2021, 3:26 PMAre there any errors or anything else suspicious in the output from dagster-daemon run?
daniel
03/04/2021, 3:29 PMI'm not really clear on why there would be so many dagster-daemon run processes in that htop - just confirming, it's set up in Google Cloud to only run a single dagster-daemon process?
daniel
03/04/2021, 3:35 PMparticularly the fact that many of them only run for a couple of seconds is confusing - it's supposed to be a single long-running process, could something in google cloud be repeatedly trying to restart it for some reason?
j
João Luiz Carabetta
03/04/2021, 3:49 PMdeamon logs look fine
João Luiz Carabetta
03/04/2021, 3:50 PMwe just ran one process
João Luiz Carabetta
03/04/2021, 3:51 PMthe google cloud instance just has dagster and dagit installed
João Luiz Carabetta
03/04/2021, 3:51 PMnothing else
d
daniel
03/04/2021, 3:51 PMhmmm, any idea where all those other dagster-daemon run could be coming from? I'm not an htop expert, but I'm not seeing that when I run the daemon locally in 0.10.7 and run htop
daniel
03/04/2021, 3:53 PMAnd I can't think of anything in dagster that automatically kicks off a "dagster-daemon run" process
j
João Luiz Carabetta
03/04/2021, 3:55 PMwe are running the daemon through a Makefile, https://github.com/RJ-SMTR/maestro/blob/main/Makefile
João Luiz Carabetta
03/04/2021, 3:55 PMcan this be a problem?
d
daniel
03/04/2021, 3:57 PMWhat triggers a 'make run-daemon' call? If that's running automatically in some way, that could explain this.
j
João Luiz Carabetta
03/04/2021, 3:59 PMit is manually trigged
João Luiz Carabetta
03/04/2021, 3:59 PMthere are a lot of grpc jobs as well
João Luiz Carabetta
03/04/2021, 3:59 PMJoão Luiz Carabetta
03/04/2021, 4:00 PMI am running without Makefile and it is still creating dozens of daemons
d
daniel
03/04/2021, 4:01 PMSo just to confirm the repro steps: You run "dagster-daemon run", without the Makefile, run htop, and then see dozens of "dagster-daemon" prcoesses?
daniel
03/04/2021, 4:07 PMwhen you start up a new dagster-daemon process, does the deployment script do anything to ensure that the previous one stops?
j
João Luiz Carabetta
03/04/2021, 4:16 PMyes for the first question
João Luiz Carabetta
03/04/2021, 4:16 PMno for the second. What should I do? It just runs through the pipeline
d
daniel
03/04/2021, 4:17 PMhow strange - does that also happen to you if you run dagster-daemon locally? or just in Google Cloud?
daniel
03/04/2021, 4:20 PMAh, so just backing up a sec - how often are you running 'Make run-daemon'? Are you running it to manually kick off the dagster pipeline?
daniel
03/04/2021, 4:25 PMIf you have a single pipeline that you want to manually trigger via command line, then I think what you want is the 'dagster pipeline launch' CLI command (or to launch the pipeline from dagit). You don't actually need the daemon (or a schedule) at all in this case. This is one of the nice things about dagster compared to, say, Airflow - you don't actually need a schedule or a scheduler at all if you want to manually trigger a pipeline.
daniel
03/04/2021, 4:28 PMI'd be happy to hop on a quick call to discuss too if that's easier
daniel
03/04/2021, 5:05 PM(You can also launch the pipeline manually from dagit if you have dagit running - that also doesn't require deploying the daemon unless you want some advanced features like sensors or limiting the number of runs happening at once)
l
Laura Moraes
03/04/2021, 5:07 PMhi daniel, we have daemon on because we're running a data acquisition process every minute
Laura Moraes
03/04/2021, 5:12 PMcan we hop on a quick call?
d
daniel
03/04/2021, 5:14 PMdefinitely - is now good?
daniel
03/04/2021, 5:15 PMI can DM you
daniel
03/04/2021, 5:33 PM(just following up here quickly - laura showed me the tmux output that shows a dangling thread, I'm going to try to reproduce that locally and will follow up here. That sounds like a different thing than the 'multiple daemon processes' topic that I was discussing with Joao)