https://dagster.io/ logo
#ask-ai
Title
# ask-ai
j

Jérôme MONTIGNY

02/22/2024, 3:49 PM
I have a dbt project with tests on sources wirtten in a file named sources.yml in a folder in the dbt models' folder. When I run my job, these tests aren't run, and I'm not getting why. Here is how my job is defined :
Copy code
@dbt_assets(manifest=dbt_manifest_path)
def dbt_project_name_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build"], context=context).stream()

dbt_project_dir = (
    Path(__file__).joinpath("..", "..", "..", "dbt_project_name").resolve()
)
dbt = DbtCliResource(
    project_dir=os.fspath(dbt_project_dir),
    profiles_dir=dbt_project_dir / "..",
)

defs = Definitions(
    assets=[
        dbt_project_name_dbt_assets,
        some_airbyte_assets
    ],
    schedules=schedules,
    resources={
        "dbt": DbtCliResource(project_dir=os.fspath(dbt_project_dir)),
    },
)

dbt_selection = build_dbt_asset_selection(
    [dbt_project_name_dbt_assets], dbt_select="fqn:folder_in_models"
)

my_selection = dbt_selection.upstream().required_multi_asset_neighbors()

# Define the job
my_job = define_asset_job("job_name", my_selection)
In the logs it is said it runs a dbt build command, but the dbt build command only contains the models in the folder and no test at all
s

Scout

02/22/2024, 3:50 PM
Based on the documentation, when you use the
@dbt_assets
decorator with a
manifest
argument and invoke
dbt.cli(["build"], context=context).stream()
, it should run the
dbt build
command for your dbt project. The
dbt build
command in dbt is designed to run models, tests, and snapshots in your dbt project. However, there are a few things to consider that might affect whether your source tests are being run: 1. Selection Syntax: The
dbt_select
argument you're using with
build_dbt_asset_selection
specifies
"fqn:folder_in_models"
. This selection syntax might be excluding your source tests if they are not within the specified folder or do not match the fully qualified name (fqn) pattern. You need to ensure that the selection criteria include the tests you want to run. 2. Manifest File: The
manifest
file should be up-to-date and include the definitions for your source tests. If the manifest file is not correctly generated or is outdated, it might not include the source tests, and therefore, they won't be run. 3. dbt Project Configuration: Ensure that your dbt project is correctly configured to recognize the
sources.yml
file and that the tests are properly defined within it. If there's a configuration issue, dbt might not be aware of the tests to run them. 4. dbt Version: Make sure that the version of dbt you are using supports the behavior you expect. Different versions of dbt might have different behaviors or bugs that could affect the execution of tests. If you've checked these points and the tests are still not running, you might want to manually run the
dbt build
command outside of Dagster with the same selection criteria to see if the tests are executed. If they run as expected outside of Dagster, there might be an issue with how Dagster is invoking the dbt command or with the integration between Dagster and dbt. The documentation does not provide a specific reason why your source tests would not be run if the
dbt build
command is correctly executed. If you continue to face issues, you may want to look into the logs more closely, check for any errors or warnings, and ensure that the dbt project is set up correctly. If necessary, you can also reach out to the Dagster community or support for further assistance.
j

Jérôme MONTIGNY

02/22/2024, 4:00 PM
@Scout I am not sure about everything, but when I run the dbt build -s "fqn:my_folder_name" , the tests are run
s

Scout

02/22/2024, 4:00 PM
If running
dbt build -s "fqn:my_folder_name"
from the command line executes the tests as expected, then it's likely that the issue lies with how the Dagster job is configured or how it invokes the dbt command. Given that the command line invocation works, here are a few steps you can take to debug the issue within your Dagster job: 1. Check the Manifest: Ensure that the manifest file being used by Dagster (
dbt_manifest_path
) is the same one that is generated when you run the dbt command from the command line. If they are different, the Dagster job might be using an outdated or incorrect manifest. 2. Review Dagster Job Configuration: Double-check the configuration of your Dagster job, particularly the
dbt_selection
and
my_selection
parts. Make sure that the selection criteria used in the job match the ones you use on the command line. The
build_dbt_asset_selection
function and the
upstream().required_multi_asset_neighbors()
method should be configured to include the tests in the selection. 3. Examine the Logs: Look at the logs generated by the Dagster job to see if there are any warnings or errors related to dbt test selection or execution. The logs might provide clues as to why the tests are not being run. 4. Test with a Simplified Job: Try simplifying your Dagster job to just run the dbt command without any selection criteria to see if the tests run in that case. If they do, gradually add back the selection criteria to identify at which point the tests stop being executed. 5. Dagster Version: Ensure that you are using a version of Dagster that supports the expected behavior with dbt. According to the documentation, starting from Dagster version 1.4.0, the default command for materializing dbt assets is
dbt build
, which includes running tests. If you are using an older version, consider upgrading. 6. Dagster and dbt Integration: Review the documentation for any updates or changes in the integration between Dagster and dbt that might affect test execution. If after these steps the issue persists, you may want to create a minimal reproducible example and reach out to the Dagster community or support with specific details about your setup and the behavior you're observing. They might be able to provide more targeted assistance.
j

Jérôme MONTIGNY

02/23/2024, 3:59 PM
@Scout it does work when I do it without a dbt selector, however, when I add a dbt selector, even though the selector would select my tests when doing a dbt list or a dbt build, Dagster doesn't run it. Is there a specific behaviour in my Dagster code that doesn't run test when I use a dbt_select in build_dbt_asset_selection ?
s

Scout

02/23/2024, 3:59 PM
Based on the documentation, the
build_dbt_asset_selection
function is used to build an asset selection for a dbt selection string. This function should translate the dbt selection criteria into a selection of Dagster assets, which includes both dbt models and tests. If the tests are not running when you use a
dbt_select
in
build_dbt_asset_selection
, it could be due to a few reasons: 1. Selection Criteria: The
dbt_select
string you provide to
build_dbt_asset_selection
might not be including the tests. Ensure that the selection criteria you use are broad enough to include both the models and the tests you want to run. 2. Dagster Behavior: There might be specific behavior in Dagster's dbt integration that affects how tests are selected and run. For example, Dagster might be interpreting the selection criteria differently than dbt does, leading to tests being excluded from the run. 3. Version Compatibility: Ensure that you are using a version of Dagster that correctly supports the dbt selection syntax and the expected behavior of running tests. As mentioned earlier, starting from Dagster version 1.4.0, the default command for materializing dbt assets is
dbt build
, which should include running tests. 4. Integration Issues: There could be an issue or a bug in the integration between Dagster and dbt that is causing this behavior. You can check Dagster's GitHub issues and discussions to see if others have encountered similar problems and if there are any known workarounds or fixes. If you have confirmed that the selection criteria are correct and the issue persists, it may be helpful to reach out to the Dagster community or support with specific details about your setup, the version of Dagster you are using, and the behavior you're observing. They may be able to provide more targeted assistance or identify if there is a bug that needs to be addressed.
9 Views