I have a custom io-manger with a `._get_s3_url(con...
# ask-community
I have a custom io-manger with a
method. More-or-less, I'm using the url to build a table structure in an s3 bucket. Is there a way to easily replicate dagster's context behavior for assets in a query library without manually specifiying which assets have partitions etc?_
Ideally, I'd like to just be able to import my io-manager in a query library and then just call it with an asset name and have the url resolve for me.
seems like maybe
will do this for me?
but that seems to limit me to exactly the asset that I've materialized. I might, for example, want to query a subset of a table
Hi! When you refer to a query library, are you talking using the query library as part of the custom I/O manager or outside of Dagster? ie. rolling your own custom trino I/O manager that'll read and write to S3 for you in a tabular format?
something outside of dagster completely.
more or less, I create a filestructure in dagster dispatching off of the Input/OutputContext
I would like to be able to export that to a user who might like to query our data
the io-manager follows the same pattern as i.e.
w/ the single function to generate a path
Aaah, okay. If that's the case, would you be able to access the repositorydef/dagster instance/etc., look-up the asset defs, and get the partition ranges from there?
would it be possible to write an asset that exports the defs?
I'd like to be able to run the library from an ipython notebook, I could guarantee access to dagster, but I'd prefer not having to.
So, I'm trying to build this using
. It doesn't look like dagster really stores much info about the assets here. Ideally, I'd like a function
asset_key -> AssetDefinition, group_name
from inside an