https://dagster.io/ logo
#ask-community
Title
# ask-community
s

Shriram Holla

05/25/2022, 8:26 PM
Hi, We use build systems like bazel to compile all our code. Ideally, we would like to use bazel to run each of our jobs. Right now we do this by using
execute_in_process
. However, this doesn’t allow us to have an interactive UI. Is there a way to use dagit while running jobs through a build system?
🤖 1
r

rex

05/25/2022, 8:34 PM
Yes this is possible. •
execute_in_process
takes in an
instance
argument. The instance can be created with a
dagster.yaml
. This configuration file defines where the process should be sending its structured event metadata to. (docs here: https://docs.dagster.io/deployment/dagster-instance) • Dagit also uses an
dagster.yaml
to know what database to use for fetching information. • So, if you have a hosted Dagit that uses the same
dagster.yaml
that you are using in your bazel build system, you should be able to visualize the running jobs.
s

Shriram Holla

05/25/2022, 8:49 PM
This is very useful! What about running jobs through dagit? This might be a stretch but is it possible to start a job through a build system on dagit? I want to be able to specify how to run a given job through dagit.
r

rex

05/25/2022, 8:53 PM
Dagit will launch jobs in isolated processes - if the isolated process has all the dependencies and information to run the bazel job, then yeah it should be possible. e.g., using the kubernetes run launcher, Dagit will launch the jobs in Kubernetes job pods with specified images. In your case, the images need to have bazel installed, the code it needs to compile, etc
s

Shriram Holla

05/25/2022, 8:59 PM
But how this configured? Eg: How would dagit know that to run JobA, it needs to run
bazel run JobA
, irrespective of where dagit is deployed
r

rex

05/25/2022, 9:12 PM
i was assuming that step would be an op in your job
d

daniel

05/25/2022, 9:45 PM
Hi Shriram - you can use bazel to build your code into a py_binary (we've had users do that before) that runs the dagster CLI in your python environment - but dagit expects that it will be able to run the 'dagster' command in order to be able to run the dagster CLI in your code to launch runs
Everytime a run is launched in dagit, it makes a 'dagster api execute_run' command and assumes that points as an entry point to the dagster CLI
s

Shriram Holla

05/26/2022, 7:52 PM
thanks for the response. we’re doing something similar with airflow by running tasks through bash operators. we were hoping to move away from that.
85 Views