Hi, I am using dagster-shell solid to create a python virtual environment and install packages into it . Is there a way, I can see the running log of the package installation?

Hi Navneet - they're likely in the compute logs for the step. The compute logs are the raw stderr / stdout for the step process (as opposed to the Dagster logs, which are structured logs). To view them, if you're on the run page, there's a button right above the panel that shows the log entries, on the left side. Does that work for you?

@Navneet Sajwan I've been using

subprocess.Popen

to monitor stdout of a long-running external program (see below). It's pretty crufty though, is there a better way @sandy?

Hi @sandy , I can some compute logs there. I don't understand why can't I see installation logs.

one thing to try: shell solids/ops accept an "output_logging" config option. here's what it accepts:

"output_logging": Field(
            Enum(
                name="OutputType",
                enum_values=[
                    EnumValue("STREAM", description="Stream script stdout/stderr."),
                    EnumValue(
                        "BUFFER",
                        description="Buffer shell script stdout/stderr, then log upon completion.",
                    ),
                    EnumValue("NONE", description="No logging"),
                ],
            ),
            is_required=False,
            default_value="BUFFER",
        ),

you could do

my_shell_solid_with_stream_logging = my_shell_solid.configured({"output_logging": "STREAM"})

hi @sandy, where do i add this part,

"output_logging": Field(
            Enum(
                name="OutputType",
                enum_values=[
                    EnumValue("STREAM", description="Stream script stdout/stderr."),
                    EnumValue(
                        "BUFFER",
                        description="Buffer shell script stdout/stderr, then log upon completion.",
                    ),
                    EnumValue("NONE", description="No logging"),
                ],
            ),
            is_required=False,
            default_value="BUFFER",
        ),

you don't need to add that snippet anywhere - I just included that for documentation. you would add:

my_shell_solid_with_stream_logging = my_shell_solid.configured({"output_logging": "STREAM"})

Thanks @sandy, it works with a little modification. Need to pass the name parameter:

my_shell_solid_with_stream_logging = my_shell_solid.configured({"output_logging": "STREAM"}, name="solid_name")