Hey everyone. I'm hoping someone can point me towards some examples and best practices for using Dagster with other languages (R in particular). I have several ETL pipelines written entirely in R that could really benefit from using a data orchestrator and Dagster is my first choice. I'm trying to find a balance between rewriting my entire process in Python versus just calling a bunch of R scripts sequentially. Any insight here is appreciated!

hi @Philip Orlando ! one middle-ground solution would be to treat your R steps as shell commands (https://docs.dagster.io/_apidocs/libraries/dagster-shell). The benefits of this approach would be that you don't need to rewrite your logic in a new language (and all of the benefits of orchestration). The cons would be: • You would still need to define dependencies between these steps in python (which might make it harder to understand where data is coming from if you're just looking at R) • You can't use dagster-specific constructs (like the IOManager, which marshals data between steps) within your transformation code In general, there are many abstractions that may be useful to you when writing your transformation logic (like resources, the

context

parameter, the afore-mentioned IOManager) which you will only have access to when writing in Python. So I'd say when writing new code, you do miss out on these benefits if you write in a different language and just invoke it as a script. But for pre-existing code that you want to orchestrate, I think this would be a reasonable solution

Hi @owen, thank you so much for providing this info! The dagster-shell doc is very helpful. I'm also going to explore using rpy2 as a workaround for some of the cons that you mentioned.