@Scout I want to define in one place all my airbyte assets that I will be reusing across different pipelines. I thought of building a custom python package, common/ to include those assets, then load the assets in other pipelines by importing the common package. Is this a bad idea ?

We encourage reusability and modularity in your Dagster projects, and defining your Airbyte assets in a common location to be reused across different pipelines is a good practice. This approach aligns with the principles of software engineering where code reuse can lead to more maintainable and scalable systems. Creating a custom Python package, such as

common/

, to include those assets can be an effective way to organize your code. You can then import and load the assets in other pipelines by referencing this common package. This is similar to how you might handle any other shared Python code. The documentation provides examples of how to define and load Airbyte assets using the

dagster_airbyte

module. For instance, you can define Airbyte assets using the

build_airbyte_assets

function and then load them into your Dagster definitions:

python
from dagster_airbyte import build_airbyte_assets, AirbyteCloudResource
from dagster import Definitions, EnvVar

airbyte_instance = AirbyteCloudResource(
    api_key=EnvVar("AIRBYTE_API_KEY"),
)
airbyte_assets = build_airbyte_assets(
    connection_id="your_connection_id",
    destination_tables=["your_destination_tables"],
)

defs = Definitions(assets=[airbyte_assets], resources={"airbyte": airbyte_instance})

You can place such code within your

common/

package and then import

defs

in other pipelines to use the defined Airbyte assets. Additionally, you can use utilities like

load_assets_from_package_module()

to load all assets defined within a Python package, which can further simplify the process of reusing assets across different pipelines. Remember to manage dependencies carefully and ensure that the common package is accessible in the environments where your pipelines run. This might involve setting up proper Python paths, using virtual environments, or installing the package in your deployment environment. In summary, your idea of building a custom Python package to include and reuse Airbyte assets across different pipelines is a sound approach and is supported by the patterns and utilities provided by Dagster. Using Dagster with Airbyte Cloud button Add upstream option to load_assets_from_... button Dagster Docs button Understanding use of assets across code ... button Thank you for your feedback!

@Scout here is my folder structure: dagster_cloud.yaml setup.py dagster/ common/ pipelines/ pipeline_1/ pipeline_2/ How do I ensure the common package is added as a dependency of the pipeline_1 package ? Give me the syntax of the setup.py file

To ensure that the

common

package is added as a dependency of the

pipeline_1

package, you will need to structure your

setup.py

file to include both packages and define the dependency relationship. Here's an example of how you might structure your

setup.py

file:

python from setuptools import setup, find_packages  setup(     name='dagster_project',     version='0.1.0',     description='A Dagster project with common assets and pipelines',     author='Your Name',     author_email='<mailto:your.email@example.com|your.email@example.com>',     packages=find_packages(),     install_requires=[         # List your project's dependencies here.         # If `common` has external dependencies, list them here as well.     ],     # If `common` is used by `pipeline_1` internally and not installed separately,     # you don't need to specify it in `install_requires`. Just ensure it's discoverable     # as part of your project's packages. )

In this

setup.py

file,

find_packages()

will automatically discover all packages in your project, including

common

and

pipeline_1

. Since

common

is part of the same project and not an external dependency, you don't need to list it in

install_requires

. The

install_requires

section is typically used for external dependencies that need to be installed from PyPI or another package index. If

common

is a namespace package that is shared across multiple projects and you want to install it separately, you would need to create a separate

setup.py

for

common

and then install it as an external dependency for

pipeline_1

. However, based on your folder structure, it seems that

common

is intended to be an internal package within the same project. Make sure that your Python environment is set up correctly and that you install your

dagster_project

package in this environment. If you're using a virtual environment, you can install the package in editable mode with

pip install -e .

from the root directory where

setup.py

is located. This will allow you to import

common

from within

pipeline_1

or any other part of your project.