https://dagster.io/ logo
Title
g

Guy McCombe

11/30/2022, 4:25 PM
Hi all! Just wondering what’s the best way to include the
CacheableAssetsDefinition
object generated by the
load_assets_from_airbyte_instance
function into a repository? At the moment we have it defined like so:
definitions = [
        airbyte_assets,
        with_resources(all_assets, resources),
        *all_jobs
    ]

    return definitions
However the inclusion of the
CacheableAssetsDefinition
(here
airbyte_assets
) converts the repository to a
PendingRepositoryDefinition
object, which doesn’t feature the methods that we’re using to test the repository (e.g.
get_all_jobs
). Any tips? Thanks! 😁
a

Adam Bloom

11/30/2022, 5:06 PM
here is what I have in my user code:
airbyte_assets = with_resources(
        airbyte_cacheable_assets.build_definitions(airbyte_cacheable_assets.compute_cacheable_data()),
        {"airbyte": airbyte_configured},
    )
:daggy-love: 1
:ty-spinny: 1
s

Sven Lito

12/16/2022, 12:53 PM
@Adam Bloom could you extend the example to show imports etc. ?
a

Adam Bloom

12/16/2022, 2:42 PM
I’d use https://docs.dagster.io/integrations/airbyte over any example I can quickly pull from my user code. I can post a more complete asset snippet if useful, but it doesn’t have any usage of the assets (jobs, repository, etc) that you can all find in the more complete example above
s

Sven Lito

12/16/2022, 6:01 PM
@Adam Bloom Thanks for the ref but I've got airbyte working fine. Just when I try
get_all_jobs
tests like Guy above I get a
PendingRepositoryDefinition
so I thought I'd ask.
g

Guy McCombe

12/19/2022, 7:38 AM
FWIW we’ll be moving to the new airbyte ingestion as code structure which I would presume doesn’t produce cacheable assets as it won’t depend on the airbyte resource to create the assets at runtime