https://dagster.io/ logo
#ask-community
Title
# ask-community
o

Ohad

02/05/2023, 7:53 PM
Hey! I'm trying to create the following, Shell script asset that starts an Airbyte asset which calls a dbt asset. Like in the screenshot. When I materialize the asset, the shell scripts complete successfully, it creates a CSV file and places it within
/tmp/airbyte_local/
folder. But, the following asset materialization step fails, complaining the file is missing. However, when I run the Airbyte sync job manually from Airbyte UI it works. According to the error, I am missing io manager, but I'm not sure what kind I should add and where. Also, why it would require to use io manager in this case?
Hey @ben I think you might know the answer to this. Any chance you have a look please?
b

ben

02/08/2023, 4:24 PM
Hi Ohad,
load_assets_from_airbyte_instance
produces a list of assets, so no need to wrap it in
attendance_shell_asset
here you could just do
Copy code
attendance_shell_source_asset = …

attendance_shell_asset = load_assets_from_airbyte_instance(...)
o

Ohad

02/09/2023, 12:38 AM
Hi @ben, I might be wrong but I think you missed my point. I need to connect a shell script (this asset extracts some attendance data to CSV) to an Airbyte asset. If I do as you suggested, I will lose the data extraction node, see screenshot.
b

ben

02/09/2023, 1:21 AM
I see, this makes sense. I think that this is a gap in functionality right now - we don’t have a way to add an upstream asset of the Airbyte generated assets. This is something we could add pretty easily though
Actually, if you use the
build_airbyte_assets
API instead of
load_assets_from_airbyte_instance
, you can specify a list of
upstream_assets
o

Ohad

02/09/2023, 2:23 AM
Good idea, I will give it a go. Thanks!
Should I also open Github issue for adding this functionality in
load_assets_from_airbyte_instance
?
b

ben

02/09/2023, 4:21 PM
Yes, that’d be great
Hi @ben, as suggested I'm trying to implement
build_airbyte_assets
, but I can't figure out how to add the input for the
upstream_assets
? The requirement is
Set[AssetKey]
, but it's unclear what this should include? Also, according to the docs, I think I need to also have
normalization_table
included, but not sure how. The screenshot is just one of the many intents I made to add the upstream asset.
b

ben

02/13/2023, 9:32 PM
Hi Ohad, `upstream_assets`should just take the asset key for
extractStudentsAttendance
, which in this case is just the string representation of the function name:
upstream_assets={AssetKey("extractStudentsAttendance")}
normalization_table
can just be empty in this case
o

Ohad

02/13/2023, 11:46 PM
Thanks @ben, I tried your suggestion, but I'm getting the attached error.
b

ben

02/13/2023, 11:47 PM
Hi Ohad, can you show me where you’re using
attendance_shell_asset
?
o

Ohad

02/13/2023, 11:48 PM
Sure, I'm using it in the list of assets within the definition
b

ben

02/13/2023, 11:49 PM
I think you’ll need a
*
in front of
attendance_shell_asset
here, since
with_resources
returns a list
o

Ohad

02/13/2023, 11:56 PM
Okay, that worked! Thanks for that! The Airbyte asset is working now! However, one issue, how can I re-connect the Aitbye asset to the downstream dbt asset now?
b

ben

02/13/2023, 11:59 PM
If you click on the asset above the dbt asset there, can you show me what the sidebar shows? That is the “asset key” that dbt is looking for, and it looks like it doesn’t match the asset key for our Airbyte asset
o

Ohad

02/14/2023, 12:05 AM
Sorry, but it doesn't open any sidebar when I click on it..
b

ben

02/14/2023, 12:09 AM
There should be a right-hand bar clicking on
src_webattend_attendance
, like the following - are you not seeing that?
o

Ohad

02/14/2023, 12:11 AM
I can see this menu within the Deployment tab, like in this example.
but I don't get this sidebar on the Assets tab.
If I click the Definition for the dbt downstream model, I get this.
b

ben

02/14/2023, 12:13 AM
If you click on the upstream asset there, what does it show?
If we can get the full asset key that DBT is looking for, we can make sure the Airbyte asset key matches
o

Ohad

02/14/2023, 12:14 AM
I get this
b

ben

02/14/2023, 12:14 AM
aha, perfect so we will want to add this to the `build_airbyte_assets`:
asset_key_prefix=["sentral_sources"]
this is so our airbyte asset has the
"sentral_sources"
prefix, you can see that here
o

Ohad

02/14/2023, 12:15 AM
oh okay! let me try that
Perfect! Thank you!! We are good now 🙂
b

ben

02/14/2023, 12:20 AM
Amazing! Thanks for your patience 🙌
2 Views