https://dagster.io/ logo
#ask-community
Title
# ask-community
s

Son Giang

03/25/2022, 4:28 AM
Hi all, what is the best way to setup Dagster for multi-tenants use case? To be easier to understand, let looks at this example. We have 3 tenants: Tenant1, Tenant2 Tenant3 We have this DAG : opA -> opB -> opC opA -> opD -> opE We want to use this same DAG for all 3 tenants, but with different configuration for each tenant. Then I want the Dagit UI to show the jobs like this: Tenant1_DAG: … Tenant2_DAG: … Tenant3_DAG: … I hope that is clear enough. Can we do things like this example?
j

jamie

03/25/2022, 1:20 PM
Hi @Son Giang! If I understand your use case correctly, I believe you should be able to make a graph of your three ops, then create multiple jobs from that graph using the
<http://GraphDefinition.to|GraphDefinition.to>_job
function. You can pass unique resource config and run config to the
to_job
function. In Dagit, the jobs will get the names you provide to them in
to_job
. So your code might look something like
Copy code
@graph 
def my_graph():
    opA()
    opB()
    opC()

tenant1_dag = my_graph.to_job(resource_defs, run_config, ...)
tenant2_dag = my_graph.to_job(resource_defs, run_config, ...)
tenant3_dag = my_graph.to_job(resource_defs, run_config, ...)
In dagit you should see three jobs
tenant1_dag
,
tenant2_dag
,
tenant3_dag
https://docs.dagster.io/concepts/ops-jobs-graphs/jobs-graphs#from-a-graph https://docs.dagster.io/_apidocs/graphs#dagster.GraphDefinition.to_job
👍 1
s

Son Giang

03/26/2022, 12:33 AM
Hi @jamie, thank you for the help, that’s what I need. However, can you tell me what is the best practice for loading configuration from database and use that for creating job? Should it be defined in
__init__.py
in jobs folder? And can it be reused from resource/config inside the op/job? Like I have a database connection that contains both the configuration to create a job and the run config of a job.
262 Views