Hi all! Has anyone attempted to use dagster with ibis? https://ibis-project.org

I've used Ibis before and it's awesome, but haven't used it with Dagster before. I'm gonna cc: @jamie, not because she needs to answer, but as a heads up in case she wants to get the gears in her head turnin'

I’m happy to help contribute here if there’s something I can do 🙂

this is very cool! At the very least i could see a resource that handles connection creation being a useful integration. I’d need to think a bit more about how an io manager would work. In the meantime, there’s nothing stopping you from instantiating an ibis connection directly in ops/assets and using it. You just might need to rely more heavily on

non_argument_deps

since you’d be handling IO directly in the asset

@jamie as a starting point, would making an ‘ibis_pandas’, ‘ibis_polars’ resource and running ibis in interactive mode make sense? Then it would be basically the same as using those tools today but with writing transformation logic in a common sql dialect.

perhaps! I’m not familiar with ibis at all so i dont really know what that would look like. If you’re interested and want to put up a PR I’m happy to review + give early feedback as you work on it