https://dagster.io/ logo
Powered by
#dagster-support
Title
# dagster-support
g

Gabriel Montañola

03/31/2022, 10:44 PM
hi there folks! I'm trying to use Software Defined Assets in order to: 1. Launch a Airbyte sync 2. Run dbt models for Airbyte raw data Things are kind of working. The Airbyte connection syncs a bunch of tables that are also exposed on 
destination_tables
but I get this after a run.
Copy code
Solid airbyte_sync_69021 did not fire outputs {'sku_availability', 'raw_product', 'raw_product_packging_variables', 'raw_product_maturation_variables', 'raw_product_provider', 'raw_product_quality_variables', 'selling_price', 'customer', 'coupon', 'sku', 'buying_price'}
These tables do not generate outputs and I don't know why. But others in the same connection work just fine. Any clues about where might the problem be? I'll provide a code snippet on the thread.
Copy code
from dagster import AssetGroup
from dagster.utils import file_relative_path
from dagster_airbyte import build_airbyte_assets
from dagster_dbt import load_assets_from_dbt_project
from ..resources import dbt_resource, airbyte_resource

DBT_PROJECT_DIR = file_relative_path(__file__, "../../../dbt")

BRAIN_CONNECTION_ID = "xxxxx"
AIRBYTE_DESTINATION_TABLES = [about 20 tables]

airbyte_brain_assets = build_airbyte_assets(
    connection_id=BRAIN_CONNECTION_ID,
    destination_tables=AIRBYTE_DESTINATION_TABLES,
)

dbt_brain_assets = load_assets_from_dbt_project(
    project_dir=DBT_PROJECT_DIR,
    profiles_dir=DBT_PROJECT_DIR + "/profile",
    select="tag:sources,tag:brain",
)


brain_source_assets = AssetGroup(
    airbyte_brain_assets + dbt_brain_assets,
    resource_defs={
        "airbyte": airbyte_resource,
        "dbt": dbt_resource,
    },
).build_job("brain_source_assets")
o

owen

03/31/2022, 10:55 PM
hi @Gabriel Montañola! one quick clue to figure out what might be happening is to look at the set of AssetMaterialization events produced by the airbyte_sync_... step in the most recent run. Two things you might discover there: 1) there are fewer materializations than tables in AIRBYTE_DESTINATION_TABLES 2) there's an equal number of materializations.
if 1) is the case, then I'd check in airbyte to confirm that all the tables you expect to be synced really are being synced, but my guess is that 2) is more likely
the materializations that dagster outputs at runtime will be the full set of tables that airbyte is syncing, but it's possible that their names don't exactly match the names given in the list (which they will need to in order to avoid this error). if you have a table prefix defined on your airbyte connection (for example), then that should be included in the name in DESTINATION_TABLES
happy to help debug more if neither of those suggestions are fruitful
g

Gabriel Montañola

03/31/2022, 11:54 PM
heya! thanks for the assistance! I triple checked names and everything matches. My guess is that since some tables are synced with CDC, if the sync run and no records are added there is nothing to materialize, right?
o

owen

04/01/2022, 12:24 AM
ooh not really intended (my fault — i wrote it! ) thanks for tracking that down i can patch it up for next week’s release
❤️ 3
g

Gabriel Montañola

04/01/2022, 12:59 AM
thank you! that's awesome! by the way this is my first interaction with Dagster! and it's looking preeeetty good! congratz to the whole team!
m

Marjori Pomarole

04/01/2022, 11:28 AM
Thanks so much @owen, indeed would be great for us to keep using software defined assets for this use cause. So much power in tracking down states and data governance for us.
❤️ 1
o

owen

04/01/2022, 4:26 PM
no problem!
just circling back on this -- a fix for this behavior has been merged in and will be included in this week's 0.14.8 release
🚀 2
m

Marjori Pomarole

04/05/2022, 4:24 PM
You guys are fast! Amazing we can continue to use assets
🙌 3
3 Views
Powered by