https://dagster.io/ logo
#ask-community
Title
# ask-community
b

Bryce Arden

03/20/2023, 12:46 PM
Hi all! Has anyone attempted to use dagster with ibis? https://ibis-project.org
Motivation: portability between different query execution backends without logic that specializes on different DataFrame types (ref).
t

Tim Castillo

03/21/2023, 12:10 AM
I've used Ibis before and it's awesome, but haven't used it with Dagster before. I'm gonna cc: @jamie, not because she needs to answer, but as a heads up in case she wants to get the gears in her head turnin'
❤️ 1
b

Bryce Arden

03/22/2023, 3:20 PM
I’m happy to help contribute here if there’s something I can do 🙂
j

jamie

03/22/2023, 3:24 PM
this is very cool! At the very least i could see a resource that handles connection creation being a useful integration. I’d need to think a bit more about how an io manager would work. In the meantime, there’s nothing stopping you from instantiating an ibis connection directly in ops/assets and using it. You just might need to rely more heavily on
non_argument_deps
since you’d be handling IO directly in the asset
b

Bryce Arden

03/23/2023, 12:47 PM
@jamie as a starting point, would making an ‘ibis_pandas’, ‘ibis_polars’ resource and running ibis in interactive mode make sense? Then it would be basically the same as using those tools today but with writing transformation logic in a common sql dialect.
j

jamie

03/23/2023, 6:54 PM
perhaps! I’m not familiar with ibis at all so i dont really know what that would look like. If you’re interested and want to put up a PR I’m happy to review + give early feedback as you work on it
34 Views