Charles Leung
06/17/2022, 6:31 PM# Run config no longer works from passing it in. Must use launch pad
run_config_path = dagster.file_relative_path(
__file__, os.path.join(".", "run_config", "vocational_assets_job.yaml")
)
config = dagster.config_from_files([run_config_path])
vocational_assets_job = dagster.build_assets_job(
assets=[*extract_assets, *load_assets, *transform_assets],
name="vocational_assets_job",
config=config,
)
However, the config doesn't work and it still requires me to pass in via launchpad. Whats the best way to set a default config?claire
06/17/2022, 7:00 PMdefine_asset_job
instead of build_assets_job
in order to allow asset definitions to live directly on repositories. You can rewrite your job definition as follows:
vocational_assets_job = define_asset_job("vocational_assets_job", config=config).resolve(
[my_asset], source_assets=[]
)
Charles Leung
06/17/2022, 7:00 PMclaire
06/17/2022, 7:01 PMvocational_assets_job = define_asset_job("vocational_assets_job", config=config)
and dagster will make the resolve call for youCharles Leung
06/17/2022, 7:01 PMvocational_assets_job cannot be executed with the provided config. Please fix the following errors:
Missing required config entry "resources" at the root. Sample config for missing entry: {'resources': {'bigquery': {'config': {'dataset': '...'}}, 'gcs': {'config': {'gcs_bucket': '...'}}}}
Even though it's configured as you listed - config is a dictionary like this:
{'resources': {'gcs': {'config': {'gcs_bucket': 'my-bucket', 'gcs_prefix': 'my-prefix'}}, 'bigquery': {'config': {'project': {'env': 'GCP_PROJECT'}, 'dataset': 'my-dataset'}}}}
claire
06/17/2022, 8:08 PMCharles Leung
06/17/2022, 8:09 PMclaire
06/17/2022, 8:09 PMDagster Bot
06/17/2022, 8:09 PMsandy
06/17/2022, 8:26 PMsean
06/17/2022, 9:12 PM