https://dagster.io/ logo
#ask-community
Title
# ask-community
h

Harpal

04/06/2022, 4:18 PM
Hi all! I’m using software-defined assets (specifically the AssetGroup().build_job() methods) to import my data generated from DBT into my dagster graph. I would like to find a way for users of the script to provide arbitrary variables to my job (similar to the functionality provided by a config.yaml file. The
AssetGroup()
allows me to pass in
resource_defs
as shown below - but I do not know how to allow users to pass arbitrary values into the “vars” dictionary:
Copy code
my_job = AssetGroup(
    dbt_assets + csv_assets,
    resource_defs={
        "dbt": dbt_cli_resource.configured(
            {
                "project_dir": DBT_PROJECT_DIR,
                "profiles_dir": DBT_PROFILE_DIR,
                "vars": {"test_split": 0.2, "train_split": 0.5},
            }
        )
    },
).build_job()
I am able to run the
my_job
in a new script with the
my_job.execute_in_process()
method but I would like to pass new values for the
train_split
and
test_split
variables instead of hard-coding them.
Copy code
"vars": {"test_split": 0.2, "train_split": 0.5},
How can I pass values to jobs defined by
AssetGroup()
’s
resource_defs
?
c

claire

04/06/2022, 5:22 PM
Hi Harpal. You can omit the
.configured
on the DBT resource and instead provide the config at job-level. For example:
Copy code
# run config yaml
resources:
  my_dbt_resource:
    config:
      vars:
        test_split: 0.2
        train_split: 0.5
      ...
You could also provide this on the
config
argument of the job
h

Harpal

04/06/2022, 6:00 PM
Hi @claire and thanks for the suggestion! I’ve followed the docs and have configured the resource in a separate
config.yaml
file (see below):
Copy code
#config.yaml
resources:
  dbt_cli_resource:
    config:
      project_dir: "./dbt"
      profiles_dir: "./dbt/config"
      vars: {"test_split": 0.2, "train_split": 0.5}
I think I understand what you mean by removing the
.configured
DBT resource. But I don’t quite understand what you mean by “configure this at the job-level”. Do I have to pass this
config.yaml
file to the
AssetGroup()
,
build_job()
or
my_job.execute_in_process()
methods? Because it’s not clear which one of these would accept such an argument. Thanks in advance dagsir
c

claire

04/06/2022, 6:23 PM
Ahh right. I realized that we haven't enabled config on asset jobs yet, though it is a top priority for us and a feature that will come out in the next couple of months. Sorry about that
🎉 1