https://dagster.io/ logo
#feature-pipes
Title
# feature-pipes
p

Pavlo Yemelianov

03/11/2024, 6:52 PM
Hi guys. Sorry maybe it will be a little stupid question. I’ve recently came across pipes for aws Lambda how hard it will be to implement support for other cloud providers like gcp ?? as far as i understand the client code should be re-usable ? given that fact i can assume that it won’t be hard to port to another lang ? we currently have a pipeline with step that is running in gcp and written in JS and we ended up making some custom solution using rabbitmq as a message broker and a custom framework for cloud functions and i was wondering if there is a way to integrate it in more “native” way thanks !
a

alex

03/12/2024, 8:29 PM
I don’t think it would be too bad, heres all the code for the lambda integration https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes.py#L165-L242 which isn’t that much all in all
you could theoretically communicate from JS by implementing parts of the pipes library in JS https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py
p

Pavlo Yemelianov

03/13/2024, 7:58 AM
thanks @alex I’ll review that and think what can i do there
11 Views