https://dagster.io/ logo
#ask-community
Title
# ask-community
s

Sandro Vieira de Paula

02/15/2023, 2:13 PM
Hello! Can someone explain briefly if is possible to load data resulting from an extraction into a Postgres database? It is through IO Managers or something like that? Thank you!
❤️ 2
x

Xiaotian Yu

02/15/2023, 3:34 PM
I'm new here, and I'm trying to do something similar (though not from database). AFAIK, it may look like this: 1. create a custom io manager (or maybe there already exists), in which you need to write your logic of "load data" and "save data" methods. In your case that should be executing some SQLs, which should be an input of the methods(and of course the connection to postgres is required); 2. define
SourceAsset
as a "starting point", which should be your first asset that directly from your database. (I think) you may save the corresponding SQL to its metadata; 3. then assume the
SourceAsset
is used, e.g.
asset_one
is derived from it. In the custom io manager, the
load_input
method will try to get data from the source asset. (I think)
load_input
uses
context.upstream_output.metadata
to get the SQL you defined in the
SourceAsset
. 4. similarly,
asset_one
is then materialized. In
asset_one
you may put how it should be saved (e.g. some SQLs) in its
metadata
. (I think)
handle_output
uses
context.metadata
to get your SQLs. That's what I've got so far, and pls correct me if I'm wrong!
❤️ 1
r

Robert Lawson

02/15/2023, 4:06 PM
We worked on this but the complexity got to be too much so assets return
None
and writing/reading to postgres is handled by a db_client resource
would like to get a nice clean iomanager working but it was a pain
s

Sandro Vieira de Paula

02/15/2023, 4:35 PM
There is also the option to use Airbyte, like in this article: https://airbyte.com/tutorials/orchestrate-data-ingestion-and-transformation-pipelines
and of course you can simply load the data inside the asset function with psycopg or something like that