Scott Chamberlain12/02/2022, 5:01 PM
) on a remote server (e.g., on GCP), is there a way to configure where that shell command is being run with Dagster tooling, or just handle that myself? There’s
but that only seems relavant if the asset/op/etc. is doing IO
chris12/02/2022, 5:16 PM
just runs the shell command in whatever environment the op is set to execute within - you can change the execution environment for ops by setting the executor - IE, if you are using the multiprocess executor, the shell command will be run locally, but in its own process, and if you are using the kubernetes executor, you can configure that to run your op in gke. If you wanted to just run that specific shell command on a remote server, I would recommend actually performing that connection within the body of an op. As in, the op itself doesn't run on the remote server. Does that make sense?
Scott Chamberlain12/02/2022, 5:22 PM
chris12/02/2022, 5:34 PM