Scott Chamberlain
12/02/2022, 5:01 PMcreate_shell_command_op
) on a remote server (e.g., on GCP), is there a way to configure where that shell command is being run with Dagster tooling, or just handle that myself? There’s IOManager
but that only seems relavant if the asset/op/etc. is doing IOchris
12/02/2022, 5:16 PMcreate_shell_command_op
just runs the shell command in whatever environment the op is set to execute within - you can change the execution environment for ops by setting the executor - IE, if you are using the multiprocess executor, the shell command will be run locally, but in its own process, and if you are using the kubernetes executor, you can configure that to run your op in gke. If you wanted to just run that specific shell command on a remote server, I would recommend actually performing that connection within the body of an op. As in, the op itself doesn't run on the remote server. Does that make sense?Scott Chamberlain
12/02/2022, 5:22 PMchris
12/02/2022, 5:34 PM