Hi! I’m trying to figure out if our modelling for ...
# ask-community
Hi! I’m trying to figure out if our modelling for an
is correct. We need to crawl a paginated
- it returns data for an input
. We currently have
with a statically defined
with all the
we need to crawl. The problem is that data does not get written to the db using the io_manager until all pages are crawled. We’d like to have data persisted into the db for each page that was crawled (rather than having to wait until all of them were crawled for a
). Should we have modeled each page for a
as a
Hi @Abhischek Thottakara! I think in this case the easiest solution might be to avoid using the IOManager directly, and instead just manually write to the database within the body of the relevant op as soon as you finish crawling a page. Would that sort of setup work for you?
Yeah I think it could. Is this usecase something you’re considering for a future iteration? (I’d love to follow a Github Issue if you have)
There isn't currently any planned work directly in this area, and I don't believe a github issue exists at the moment -- would you mind writing something up with some more specifics of the feature(s) that'd be useful to you?
❤️ 1