Hello, I am new to Dagster - very nice tool, looks...
# announcements
m
Hello, I am new to Dagster - very nice tool, looks promising so far! I am currently battling with setting up the scheduler, so I am hoping to get some feedback. What I have done: 1) run the daemon via ‘dagster-daemon run’ - logs show no error but there is a message saying ‘Non checking for any run since no schedules have been started’ ; 2) set up a pipeline with daily_schedule decorator - dagit show a green light next to my scheduled process and the ‘next tick’ is correctly populated. That said, dagit also says ‘the scheduler daemon is not running’ (in line with what the status page shows). Any idea about what is missing/misconfigured?
d
Hi Marco - It sounds like the daemon and dagit might not be using the same database to persist information about runs and schedules. Have you set up a dagster.yaml file and set a DAGSTER_HOME environment variable?
m
yes. Before when I didn’t do it the daemon was not even starting
d
got it - would you mind posting the contents of that file?
m
sure, here it is
when I launch the daemon it says ’instance is configured with the following daemons: [‘QueuedRunCoordinatorDaemon’, SchedulerDaemon’,‘SensonDaemon’]
d
Got it. So that should use a SQLite database on the local machine for your storage, but dagit and the daemon aren't looking in the same place for some reason. A couple followups: • are the daemon and dagit running on the same machine? If so, everything should work fine. • Are you confident both dagit and the daemon have access to your DAGSTER_HOME environment variable to find this file? You restarted dagit after setting it, for example?
one clue would be if dagit is outputting something to the console about not being able to find a dagster.yaml file on startup - if that's happening, that would match some of these symptoms
m
ah! yes..the problem was that the terminal were dagit was lauched form had no knowledge of DAGSTER_HOME
thanks!
working fine now
🎉 1
1
d
Awesome, glad it's working. We should probably have dagit throw like the daemon when that happens, vs. leaving you in a weird state like that.
c
@daniel do you think these debug steps might fit into this page? https://docs.dagster.io/troubleshooting/schedules
d
Potentially yeah! Or maybe a general 'deployment troubleshooting' page? We can also make it not possible to get into this state in the first place by requiring dagit to have DAGSTER_HOME set: https://dagster.phacility.com/D6609
c
hmm, will move the convo to the diff 👍
m
guys, I have another issue now that I hope you can help with
I have made some changes to the py files and now when I run the dagster-daemon it raise an error related to a file that no longer exists - I think it somehow relates to the previous queue. the question I suppose is: how do I clear the queue?
btw, when I run the daemon it says ‘Retrieved 26 queud runs…’ followed by the FileNotFoundError
d
Could you post the stack trace of the FileNotFoundError that you're seeing?
To delete queued runs from Dagit, click on the Runs tab on the left - it should show any queued runs there, and there's a menu with the option to cancel or delete them.
m
Sorry away from computer now
Please note: I have already deleted the queue from the gui
Nothing is listed there as queued now, only as cancelled
d
ah great! Are you still running into problems?
or was clearing the queue enough to solve things?
m
Yes, the entries in the queue were already marked as cancelled
Still not working
Is there any other way to clean the queue?
d
Got it - so despite all your runs being cancelled, the daemon is still throwing an error about a missing file? That's definitely unexpected. Once we have the stack trace I should be able to understand more about what's going on
m
Is it possible that as the file is not there anymore the cleanup is somehow not effected?
d
If you don't have any queued runs, the daemon shouldn't be doing anything - so I wouldn't expect the file to matter.
m
Yes but the daemon still sees the queue: btw, when I run the daemon it says ‘Retrieved 26 queud runs…’ followed by the FileNotFoundError
I’ll send the stack trace shortly
d
Just to confirm - If you click on the Queued tab at the top in the runs tab in dagit, does it show no runs there?
(Just to rule out the case where everything on the first page of the Runs page was canceled, but runs on later pages are still queued - that would be one possible explanation for this)
m
Indeed - my bad. I’ve just realised that I was selecting only a subset
I assumed the select all was accounting for everything
d
Ah no problem, that’s good feedback that it’s not clear
m
Hi Daniel - thanks for helping with the previous queries. I am now in a much better place with my setup.
One thing I noticed is that Pycharm is having trouble with the dagster imports
specifically, although everything works properly at run time (dagit/dagster-daemon..) and in debugging, pycharm cannot find (and hence complete code) for anything with the dagster package, which I installed with pip and is correctly found in the env used by Pycharm
any idea what the problem might be?
*anything within the dagster package
c
I’m using pycharm for dev as well. For me auto complete works for everything except trying to auto complete the context/init_context variable within things like a solid/resource. So it can’t see log/resources/config and anything past that for completion but it also doesn’t throw any errors for me when inspecting the code
Autocompleting things like the decorator parameters and anything directly imported works though
m
Hi Cameron, thanks
‘auto complete the context/init_context variable’ -> if you annotate the context then autocompletion should work fine
can I ask you what dagster version you are using and how you have installed it?
also what pycharm version
c
Pycharm==2020.3.3 Dagster==0.10.2 Dagster (and all the project’s packages) are installed with virtualenv
What would you annotate the context as?
Nevermind, annotating it as
context: dagster.SystemComputeExecutionContext
works, the code inspection just throws an unresolved attribute when things like the resource keys are called like:
context.resources.token
m
Yes, that was my suggestion
Thanks for the version - I’ll try to update my pycharm and see if that helps