https://dagster.io/ logo
Title
a

Arthur

12/07/2022, 4:16 PM
Rookie question but i'm at my wits end, whats the pattern to load a database table as an asset with a bogstandard iomanager I'e tried
@asset(ins={"app_accounts": AssetIn(input_manager_key="stagecoach_io_manager")})
def ebdb_accounts(context,app_accounts):
return app_accounts
to no avail it gives this error.
dagster._core.errors.DagsterInvalidDefinitionError: Input asset '["app_accounts"]' for asset '["ebdb_accounts"]' is not produced by any of the provided asset ops and is not one of the provided sources
(edited)
z

Zach

12/07/2022, 4:28 PM
is your database table defined as an asset already, named as "app_accounts"? is the asset a part of an asset job that also contains ebdb_accounts? by bogstandard do you mean just the default IO Manager (that class doesn't know how to load assets stored in a database)?
a

Arthur

12/07/2022, 4:34 PM
class MYSQLMANAGER(IOManager):
def __init__(self, config):
self._config = config
def handle_output(self, context:OutputContext, obj):
context.add_output_metadata(
{
"preview": MetadataValue.md(obj.head().to_markdown()),
"description": MetadataValue.md(obj.describe().to_markdown()),
"num_rows": len(obj)
}
)
return None
def load_input(self, context:InputContext) -> pd.DataFrame:
table = context.asset_key.path[-1]
with warnings.catch_warnings():
warnings.simplefilter('ignore', UserWarning)
with connect_mysql(config=self._config) as conn:
sql = f"select * from {table}".format(table)
return pd.read_sql_query(sql, conn)
Ignore the handle_output I was more interested in load_input and I'd assumed i'd pass the name of the database table in the asset so it can be loaded but Im very wrong but cant work out how. If i remove the
ns={"app_accounts": AssetIn(input_manager_key="stagecoach_io_manager")}
I dont see how to tell the asset what database table to load
z

Zach

12/07/2022, 4:35 PM
so it's not even getting to the point where it's trying to load your database table, because
app_accounts
needs to be defined either as an asset or a SourceAsset. is it already defined?
:rainbow-daggy: 1
a

Arthur

12/07/2022, 4:36 PM
Ahh okay so i'd need sourceasset first
thank you
😛artydagster: 1
I thought you could have unconnected inputs
z

Zach

12/07/2022, 4:38 PM
you can for ops because you can provide op input values at configuration time. I don't know if the same is true for assets, it's unclear to me how you would provide the asset to the job at configuration time (would be interested to hear from someone else, assets aren't exactly my forte)
a

Arthur

12/07/2022, 4:42 PM
Thanks for the help I'd been spinning for hours looking through docs
your sourceasset suggestion worked too !