19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi, I’m new to Dagster, and have a couple of simpl...
# randomj
jasono
02/27/2022, 7:51 PMHi, I’m new to Dagster, and have a couple of simple questions that I can’t seem to figure out by reading the documentation.
Let’s say, there is a task (C) dependent on task (A) and (B), where A and B are independent of each other. I could write a Python script where Solid A, B will be run from C, and they all run successfully.
1. How can I instruct Dagster to run A and B concurrently?
2. If I need to re-run A and C, but not B, how do I do that without changing the script?
d
Daniel Kim
02/28/2022, 1:03 AMHey! I used Luigi some years ago. In your item #1, by default dagster will run applicable ops concurrently. In your item #2, there are a few different ways, but in dagit web UI, this is easily accomplished. You literally just select/highlight task A and in the dropdown option, there is option to just re-run task A or run Task A and any downstream task(s). That's it. Done.
j
jasono
02/28/2022, 1:34 AMCool. I will take your word for it and keep digging. 🙂 I once tried Prefect and got bitten pretty bad before I decided not to pursue it.
jasono
02/28/2022, 1:37 AMThank you!!
m
mrdavidlaing
03/03/2022, 2:48 AMBy design Dagster is a data orchestrator not a task runner. Dagster is trying to keep track of the state of the data assets being orchestrated.
Whilst what @Daniel Kim says is true and will work; I think doing this goes against the way Dagster is designed / intended to be used.
mrdavidlaing
03/03/2022, 2:48 AMTaking a step back; what is the higher level problem you are trying to solve?
Maybe there is a more “Dagster-y” way to since it?
j
jasono
03/03/2022, 2:51 AMSo following the example I gave at the top,
jasono
03/03/2022, 2:51 AMTask A is client grouping (ie. client id -> client group)
jasono
03/03/2022, 2:51 AMmore specifically, loading client group mapping is task A.
jasono
03/03/2022, 2:52 AMTask B is loading transaction data which includes client id.
jasono
03/03/2022, 2:53 AMTask C which depends on A & B, calculates commissions, etc and produce a summary report showing results by client group.
jasono
03/03/2022, 2:53 AMLet’s say client mapping changed after Task C successfully completed. We would want to reload the mapping, (A) and rerun task C, but not B.
jasono
03/03/2022, 2:54 AMHope it makes sense.
m
mrdavidlaing
03/03/2022, 7:55 PM1. Concurrency is determined by your Executor settings. You'll probably want to start by using the MultiprocessExecutor - https://docs.dagster.io/deployment/executors - and telling it how many things it is allowed to run in parallel (when the DAG allows)
mrdavidlaing
03/03/2022, 8:04 PMI think you'll just get 2 out of the box by using memoization- take a look at https://docs.dagster.io/guides/dagster/memoization
j
jasono
03/03/2022, 8:38 PMThank you!! I will take a look.