Hello! Can someone explain briefly if is possible to load data resulting from an extraction into a Postgres database? It is through IO Managers or something like that? Thank you!

I'm new here, and I'm trying to do something similar (though not from database). AFAIK, it may look like this: 1. create a custom io manager (or maybe there already exists), in which you need to write your logic of "load data" and "save data" methods. In your case that should be executing some SQLs, which should be an input of the methods(and of course the connection to postgres is required); 2. define

SourceAsset

as a "starting point", which should be your first asset that directly from your database. (I think) you may save the corresponding SQL to its metadata; 3. then assume the

SourceAsset

is used, e.g.

asset_one

is derived from it. In the custom io manager, the

load_input

method will try to get data from the source asset. (I think)

load_input

uses

context.upstream_output.metadata

to get the SQL you defined in the

SourceAsset

. 4. similarly,

asset_one

is then materialized. In

asset_one

you may put how it should be saved (e.g. some SQLs) in its

metadata

. (I think)

handle_output

uses

context.metadata

to get your SQLs. That's what I've got so far, and pls correct me if I'm wrong!

We worked on this but the complexity got to be too much so assets return

None

and writing/reading to postgres is handled by a db_client resource

There is also the option to use Airbyte, like in this article: https://airbyte.com/tutorials/orchestrate-data-ingestion-and-transformation-pipelines