Hi guys, could use some advice on dealing with a l...
# announcements
Hi guys, could use some advice on dealing with a legacy pipeline dependency.
I've got my dagit instance setup on EC2 it's been running beautifully for weeks. My next project will be migrating our oldest pipeline, an antiquated and overengineered mess. It is "orchestrated" from a local machine running Windows, this same machine runs a MSSQL database we need to extract data from.
This database powers a proprietary tool that is used by one of our departments. Sadly the only way I currently have to extract data from it is to run a CLI command with this tool. This generates some CSV files consumed by the pipeline.
Ideally I'd like this whole process to be mapped in the same dagster pipeline. Would it be possible to have a pipeline launched in my EC2 instance run a solid on this local box? I'm guessing this would probably involve having to create a custom RunLauncher.
Hmm, tricky business. I don’t believe a run launcher will help at all. The crux of the problem is how do you talk to the windows box
my first thought is to run a sftp / ssh server on the windows box and then have the solid in ec2 interact with the windows box that way
Ah, good idea. Thanks @alex 😁