Hey, we are managing our Spark clusters “manually”, so are ending up using

create_shell_command_op

to submit a Spark job. Since we want to create the command dynamically during execution we end up using

shell_op

, however the documentation is missing there. Is this the way to go? Any chance there is documentation of the

shell_op

? Thanks in advance to the always helpful Dagster team 🙏

Hi Pieter, that’s an oversight that it’s missing a doc string, thanks for catching that. The code for

shell_op

is here: https://github.com/dagster-io/dagster/blob/7207a6e2dc3fd3a6e9705ca361b9f5a18204c1e[…]/python_modules/libraries/dagster-shell/dagster_shell/solids.py

Thanks a lot @johann!

@Dagster Bot docs shell_op has no API docs

Created issue at: https://github.com/dagster-io/dagster/issues/7989

Hi @johann do you have an example of how the

shell_op

should be used? E.g. how the

context

arg should be defined. To give you a bit of context: I need to create a job that runs a shell command, and the shell command must be built at execution time based on parameters from a sensor

Hi Lucia, apologies for the delay. Here’s a basic example:

from dagster_shell import shell_op
from dagster import job, op

@op
def get_shell_cmd_op():
    return 'echo "hello world"'


@job
def shell_job():
    shell_op(get_shell_cmd_op())

Thanks a lot Johann!