https://dagster.io/ logo
Title
m

Martin Preusse

04/04/2023, 7:44 PM
I have a job that loads data from files to a database table. I use
ops
because I managed to use
DynamicOut
and have not figured out yet how to do parallel processing with assets/graphs 🙂 In my job, I would like to create the database table first before updating it in a parallel process:
@job
def load_hgnc_to_postgres():
    recreate_hgnc_gene_table()
    load_hgnc_file().map(load_hgnc_table)
Is it possible to wait for
recreate_hgnc_gene_table()
to finish before starting the next step?
s

sandy

04/04/2023, 8:28 PM
Can you add an input to
load_hgnc_file
and pass
recreate_hgnc_gene_table
to it? You can use a nothing dependency if you don't need to pass any data: https://docs.dagster.io/concepts/ops-jobs-graphs/graphs#defining-nothing-dependencies
m

Martin Preusse

04/05/2023, 7:28 AM
Exactly what I was looking for! Thank you so much @sandy 🚀