Hi, I'm a beginner when it comes to this, so apol...
# ask-community
j
Hi, I'm a beginner when it comes to this, so apologies if this question is a bit trivial/obvious - but I'm having a bit of trouble making the scheduler work. I won't copy the entire code below but essentially it's a football scraper -> Twitter bot. And I essentially want it to run the first op at a certain time automatically to get today's fixtures, which will then feed into the subsequent operations.
@op(
out= {
"data" : DynamicOut(),
}
)
def get Stuff():
# Scrape stuff
for idx, row in enumerate(table_rows):
# Just feed in url here and say if available for certain date (today -> then yield)
yield DynamicOutput((time, url), output_name = "data", mapping_key=f"data_{idx}")
@op
def do_Stuff(url):
# Just generate datasets then plots from url and time determines when we do so we got above
@job
def materialize_Stuff():
time, url = get_Info()
url.map(do_Stuff)
basic_schedule = ScheduleDefinition(job=materialize_view, cron_schedule="0 9 * * *")
I thought I could simply run the scheduler using that last line but seemingly not. I can see from the tutorials (https://dagster-git-docs-tutorialscheduling-elementl.vercel.app/tutorial/scheduling-your-pipeline) that they change the init file but not sure if that's relevant here (since using ops/jobs). So not really sure what to do - any help would really be appreciated. Thanks, Joe
🤖 1
s
Hi Joe, Defining a
ScheduleDefinition
or (any dagster definition) in Python is not enough by itself to execute anything-- you need to package your definitions into a
Definitions
object and run dagit against it (as in the tutorial).
j
Ah, thanks Sean. Think I was getting a bit confused since it was for asset jobs and not sure what the equivalent would be. Presumably just adding this to the end of the code will suffice (works fine by the looks of it but just checking I'm not missing anything)
basic_schedule = ScheduleDefinition(job=materialize_view,
cron_schedule="15 18 * * *",                               )
defs = Definitions(
schedules=[basic_schedule],
jobs=[materialize_view]
)
Thanks again for your help
🙌 1