https://dagster.io/ logo
Join Slack
Powered by
question for you guys about passing jobs and sched...
# ask-community
j
question for you guys about passing jobs and schedules and other stuff.. let's say i have a bunch of schedules, each one is listed in the jobs directory 
__init__
file, and then i pass that to the top-level 
__init__
file:
Copy code
from .jobs.__init__ import jobs, schedules
from dagster import Definitions

def scheduler(schedules: dict):
   for schedule_name, schedule_config in schedules.items():
       # set each schedule_name as a variable with the value of schedule_config
       # should i use exec/globals? or is there a dagster solution that is not as risky??


jobs = Definitions(jobs=[*jobs])
scheduler(schedules) --> schedule_1 = ScheduleDefinition1
                         schedule_2 = ScheduleDefinition2
any way around this using dagster
D 1
c
just a note that you don’t need to import .jobs.__init, just regular .jobs should work
what is the scheduler function supposed to do here exactly?
Having trouble parsing what you’re looking to get out of doing exactly
j
hey thanks, so basically within jobs i have a bunch of subdirectories, each one has a file or a few that end up being a job. and then i have another folder that holds other types of jobs, also having subdirectories. so my goal is to funnel all those jobs into the top-level 
__init__
doing so with jobs seems to be easy, but the schedules are tough because each one is it's variable
Copy code
from dagster import ScheduleDefinition
from .usage_from_bigq.usage_from_bigq import namea
from .usage_from_bigq.job_test import nameb

jobs = [namea, nameb]

schedules = {"basic_schedule": ScheduleDefinition(job=namea, cron_schedule="0 0 * * *", execution_timezone="US/Pacific")}
Copy code
from .jobs.__init__ import jobs, schedules
from .jobs2.__init__ import jobs as jobs2, schedules as schedules2
from dagster import Definitions

jobs = Definitions(jobs=[*jobs, *jobs2])
question is what do i do with schedules
basically id like to keep the job definitions and the schedules at the job level
c
I’m having trouble parsing the code examples you sent, is the one where you’re defining schedules living in some subdirectory and the one defining 
Definitions
is at module level?
are you saying that including schedules as a dictionary is causing the problem? If so, what problem is it causing exactly?
also, what is schedule config here? There’s no concept of configuring schedules to my knowledge in dagster
j
sorry for being confusing - im definitely not great at this! let's say i have three files: 1. a job file 2. the job directory (one of a few) with an 
__init__
3. the top level 
__init__
i want my definitions to be at level 1, to get consolidated at level 2, and then all the job directories to get consolidated at level 3 the jobs are getting imported all the way up just fine, but i'm trying to figure out a way to also initialize the schedules, because it seems that if they are not defined at level 3 then they don't work
i just got it from here
c
ah i see - since they need to import jobs
and you’ve tried including schedules in their own subdirectory?
j
yeah the dict was gonna hold a bunch of schedules...and then i was gonna make them global variables??
yeah nothing works unless its instantiated at the top init
c
What error are you getting? You might need to do an editable install of the module and change away fro doing relative imports
j
im actually not getting an error lol...its just not showing up
c
ah wait are you including the schedules in the 
Definitions
object
thats what defines what shows up in dagit
Definitions(schedules=…, …
j
ok i did not include that!
but still need to include a list of variables as an argument to that
oh wow..this works: 
schedules = [ScheduleDefinition(job=namea, cron_schedule="0 0 * * *", execution_timezone="US/Pacific")]
Copy code
jobs = Definitions(jobs=[*jobs],
                   schedules=[*schedules])
i just added a 
name=
parameter and it works
c
yup!
j
🙏
you are awesome man
c
just doin my job 🫡
🌈 1
keep sql’ing
🤣 1
Open in Slack
PreviousNext
Powered by