Simon Frid
02/24/2023, 8:17 PM@op
def small_compute_job():
return 5
@op(executor=k8s_executor)
def big_compute_job(arg):
return math.pow(arg, arg)
@job
def do_stuff():
big_compute_job(small_compute_job())
defs = Definitions(
jobs=[do_stuff], executor=multiprocess_executor
)
even more so, it would be great to be able to define some computation context for an asset
@asset(executor=my_custom_high_cpu_k8s_executor)
def a_very_big_table(arg):
return some_hyper_parallelized_tasks(arg, n=1000)
Simon Frid
02/24/2023, 8:17 PMdaniel
02/24/2023, 8:20 PMSimon Frid
02/24/2023, 8:38 PMSimon Frid
02/24/2023, 8:38 PM