https://dagster.io/ logo
#ask-community
Title
# ask-community
s

Scott Chamberlain

12/02/2022, 5:01 PM
I’ve searched through Dagster docs and googled and haven’t found anything: When a asset or op, etc. is only running a shell command (e.g.
create_shell_command_op
) on a remote server (e.g., on GCP), is there a way to configure where that shell command is being run with Dagster tooling, or just handle that myself? There’s
IOManager
but that only seems relavant if the asset/op/etc. is doing IO
c

chris

12/02/2022, 5:16 PM
create_shell_command_op
just runs the shell command in whatever environment the op is set to execute within - you can change the execution environment for ops by setting the executor - IE, if you are using the multiprocess executor, the shell command will be run locally, but in its own process, and if you are using the kubernetes executor, you can configure that to run your op in gke. If you wanted to just run that specific shell command on a remote server, I would recommend actually performing that connection within the body of an op. As in, the op itself doesn't run on the remote server. Does that make sense?
s

Scott Chamberlain

12/02/2022, 5:22 PM
Okay, makes sense. Yeah, I’m thinking the ops run on Dagster serverless, but the shell commands run on the remote GCP server. Okay, so performing connection within the op sounds like the way to go
c

chris

12/02/2022, 5:34 PM
yea I would say so!
👍 1
2 Views