Hi! I’m trying to figure out if our modelling for ...
# ask-community
a
Hi! I’m trying to figure out if our modelling for an
@asset
is correct. We need to crawl a paginated
api
- it returns data for an input
countryCode
. We currently have
@graph_asset
with a statically defined
StaticPartitionsDefinition
with all the
countryCodes
we need to crawl. The problem is that data does not get written to the db using the io_manager until all pages are crawled. We’d like to have data persisted into the db for each page that was crawled (rather than having to wait until all of them were crawled for a
countryCode
). Should we have modeled each page for a
countryCode
as a
DynamicParitionsDefinition
?
o
Hi @Abhischek Thottakara! I think in this case the easiest solution might be to avoid using the IOManager directly, and instead just manually write to the database within the body of the relevant op as soon as you finish crawling a page. Would that sort of setup work for you?
a
Yeah I think it could. Is this usecase something you’re considering for a future iteration? (I’d love to follow a Github Issue if you have)
o
There isn't currently any planned work directly in this area, and I don't believe a github issue exists at the moment -- would you mind writing something up with some more specifics of the feature(s) that'd be useful to you?
❤️ 1