https://dagster.io/ logo
Join Slack
Powered by
hi there folks! I'm trying to use Software Define...
# ask-community
g
hi there folks! I'm trying to use Software Defined Assets in order to: 1. Launch a Airbyte sync 2. Run dbt models for Airbyte raw data Things are kind of working. The Airbyte connection syncs a bunch of tables that are also exposed on 
destination_tables
but I get this after a run.
Copy code
Solid airbyte_sync_69021 did not fire outputs {'sku_availability', 'raw_product', 'raw_product_packging_variables', 'raw_product_maturation_variables', 'raw_product_provider', 'raw_product_quality_variables', 'selling_price', 'customer', 'coupon', 'sku', 'buying_price'}
These tables do not generate outputs and I don't know why. But others in the same connection work just fine. Any clues about where might the problem be? I'll provide a code snippet on the thread.
Copy code
from dagster import AssetGroup
from dagster.utils import file_relative_path
from dagster_airbyte import build_airbyte_assets
from dagster_dbt import load_assets_from_dbt_project
from ..resources import dbt_resource, airbyte_resource

DBT_PROJECT_DIR = file_relative_path(__file__, "../../../dbt")

BRAIN_CONNECTION_ID = "xxxxx"
AIRBYTE_DESTINATION_TABLES = [about 20 tables]

airbyte_brain_assets = build_airbyte_assets(
    connection_id=BRAIN_CONNECTION_ID,
    destination_tables=AIRBYTE_DESTINATION_TABLES,
)

dbt_brain_assets = load_assets_from_dbt_project(
    project_dir=DBT_PROJECT_DIR,
    profiles_dir=DBT_PROJECT_DIR + "/profile",
    select="tag:sources,tag:brain",
)


brain_source_assets = AssetGroup(
    airbyte_brain_assets + dbt_brain_assets,
    resource_defs={
        "airbyte": airbyte_resource,
        "dbt": dbt_resource,
    },
).build_job("brain_source_assets")
o
hi @Gabriel Montañola! one quick clue to figure out what might be happening is to look at the set of AssetMaterialization events produced by the airbyte_sync_... step in the most recent run. Two things you might discover there: 1) there are fewer materializations than tables in AIRBYTE_DESTINATION_TABLES 2) there's an equal number of materializations.
if 1) is the case, then I'd check in airbyte to confirm that all the tables you expect to be synced really are being synced, but my guess is that 2) is more likely
the materializations that dagster outputs at runtime will be the full set of tables that airbyte is syncing, but it's possible that their names don't exactly match the names given in the list (which they will need to in order to avoid this error). if you have a table prefix defined on your airbyte connection (for example), then that should be included in the name in DESTINATION_TABLES
happy to help debug more if neither of those suggestions are fruitful
g
heya! thanks for the assistance! I triple checked names and everything matches. My guess is that since some tables are synced with CDC, if the sync run and no records are added there is nothing to materialize, right?
o
ooh not really intended (my fault — i wrote it! ) thanks for tracking that down i can patch it up for next week’s release
❤️ 3
g
thank you! that's awesome! by the way this is my first interaction with Dagster! and it's looking preeeetty good! congratz to the whole team!
m
Thanks so much @owen, indeed would be great for us to keep using software defined assets for this use cause. So much power in tracking down states and data governance for us.
❤️ 1
o
no problem!
just circling back on this -- a fix for this behavior has been merged in and will be included in this week's 0.14.8 release
🚀 2
m
You guys are fast! Amazing we can continue to use assets
🙌 3
3 Views
Open in Slack
PreviousNext
Powered by