geoHeil
06/09/2023, 6:58 AMpython
# @op one: DBT
target = os.environ.get("DBT_TARGET", "local")
context.log.debug(f"Running dbt with target {target}")
docs_events = dbt.cli(
["docs", "generate", "--target", target, "--profiles-dir", DBT_PROFILES_DIR],
manifest=manifest,
).wait()
test_events = dbt.cli(["test"], manifest=manifest).wait()
# @op 2: Open Metadata
# requires the following items:
dbtCatalogFilePath: dbt/target/catalog.json
dbtManifestFilePath: dbt/target/manifest.json
dbtRunResultsFilePath: dbt/target/run_results.json
# @op 3: some more OM ops for different metadata i.e. metabase which also need the parsed manifest
The new API is introducing additions to the path (and the old existing ones no longer resolve).
I was thinking about calling out to the private function:
unique_target_path = dbt._get_unique_target_path()
However:
1. I am not sure if some better public alternative would be available
2. the path which is appended is different for each invocation of the DBT cli. So this means I would twice need to call this function (twice), retrieve the paths and copy/mv the files to their original directory i.e.
- for docks generate: manifest, catalog
- for test: run results
This seems fairly clumsy. Is there a better way?geoHeil
06/09/2023, 7:10 AMdbt_cli_task.get_artifact("run_results.json")
might be a bit nicer - but right now I still need the JSON files in the path so would then need to write them out again.