Hi all! Has anyone attempted to use dagster with i...
# ask-community
Hi all! Has anyone attempted to use dagster with ibis? https://ibis-project.org
Motivation: portability between different query execution backends without logic that specializes on different DataFrame types (ref).
I've used Ibis before and it's awesome, but haven't used it with Dagster before. I'm gonna cc: @jamie, not because she needs to answer, but as a heads up in case she wants to get the gears in her head turnin'
❤️ 1
I’m happy to help contribute here if there’s something I can do 🙂
this is very cool! At the very least i could see a resource that handles connection creation being a useful integration. I’d need to think a bit more about how an io manager would work. In the meantime, there’s nothing stopping you from instantiating an ibis connection directly in ops/assets and using it. You just might need to rely more heavily on
since you’d be handling IO directly in the asset
@jamie as a starting point, would making an ‘ibis_pandas’, ‘ibis_polars’ resource and running ibis in interactive mode make sense? Then it would be basically the same as using those tools today but with writing transformation logic in a common sql dialect.
perhaps! I’m not familiar with ibis at all so i dont really know what that would look like. If you’re interested and want to put up a PR I’m happy to review + give early feedback as you work on it