https://dagster.io/ logo
Join Slack
Powered by
I have a custom io-manger with a `._get_s3_url(con...
# ask-community
d
I have a custom io-manger with a 
._get_s3_url(context)
method. More-or-less, I'm using the url to build a table structure in an s3 bucket. Is there a way to easily replicate dagster's context behavior for assets in a query library without manually specifiying which assets have partitions etc?_
Ideally, I'd like to just be able to import my io-manager in a query library and then just call it with an asset name and have the url resolve for me.
seems like maybe 
RepositoryDefinition.load_asset_value
will do this for me?
but that seems to limit me to exactly the asset that I've materialized. I might, for example, want to query a subset of a table
t
Hi! When you refer to a query library, are you talking using the query library as part of the custom I/O manager or outside of Dagster? ie. rolling your own custom trino I/O manager that'll read and write to S3 for you in a tabular format?
d
something outside of dagster completely.
more or less, I create a filestructure in dagster dispatching off of the Input/OutputContext
I would like to be able to export that to a user who might like to query our data
the io-manager follows the same pattern as i.e. 
UPathIOManager
w/ the single function to generate a path
t
Aaah, okay. If that's the case, would you be able to access the repositorydef/dagster instance/etc., look-up the asset defs, and get the partition ranges from there?
d
would it be possible to write an asset that exports the defs?
I'd like to be able to run the library from an ipython notebook, I could guarantee access to dagster, but I'd prefer not having to.
So, I'm trying to build this using 
OpExectutionContext.instance.get_asset_records
. It doesn't look like dagster really stores much info about the assets here. Ideally, I'd like a function 
asset_key -> AssetDefinition, group_name
from inside an 
OpExecutionContext
Open in Slack
PreviousNext
Powered by