Joe Williamson
03/30/2023, 11:37 AM@op(
out= {
"data" : DynamicOut(),
}
)
def get Stuff():
# Scrape stuff
for idx, row in enumerate(table_rows):
# Just feed in url here and say if available for certain date (today -> then yield)
yield DynamicOutput((time, url), output_name = "data", mapping_key=f"data_{idx}")
@op
def do_Stuff(url):
# Just generate datasets then plots from url and time determines when we do so we got above
@job
def materialize_Stuff():
time, url = get_Info()
url.map(do_Stuff)
basic_schedule = ScheduleDefinition(job=materialize_view, cron_schedule="0 9 * * *")
I thought I could simply run the scheduler using that last line but seemingly not. I can see from the tutorials (https://dagster-git-docs-tutorialscheduling-elementl.vercel.app/tutorial/scheduling-your-pipeline) that they change the init file but not sure if that's relevant here (since using ops/jobs). So not really sure what to do - any help would really be appreciated.
Thanks,
Joesean
03/30/2023, 4:35 PMScheduleDefinition
or (any dagster definition) in Python is not enough by itself to execute anything-- you need to package your definitions into a Definitions
object and run dagit against it (as in the tutorial).Joe Williamson
03/30/2023, 5:28 PMbasic_schedule = ScheduleDefinition(job=materialize_view,
cron_schedule="15 18 * * *", )
defs = Definitions(
schedules=[basic_schedule],
jobs=[materialize_view]
)
Thanks again for your help