How to run dagster command line with passing YAML solid settings?

Is a resource only persistent within a solid, or can it persist from solid to solid?

I'm getting an intermittant error when dagit is starting:

Object IPCErrorMessage(serializable_error_info=SerializableErrorInfo(message="dagster.check.CheckError: Invariant failed. Description: Can only serialize whitelisted namedtuples, received tuple ('2018-01-01', None)

but I'm having a lot of trouble finding the source of the Tuple. Thoughts on where this would be coming from?

Hello, I have a question about defining inputs/outputs for

dagstermill

solids. In the docs, we see input defs defined like this:

k_means_iris = dm.define_dagstermill_solid(
    "k_means_iris",
    script_relative_path("iris-kmeans_2.ipynb"),
    input_defs=[InputDefinition("path", str, description="Local path to the Iris dataset")],
)

This way of doing things requires duplicating input definitions/descriptions between the notebook itself and the solid definition call. Ideally the input definitions could be parsed from the special

parameters

-tagged cell that you need to define anyway (some special comment formatting could maybe be used for the descriptions). A similar cell could be used for outputs. Is anything like this possible now or planned? I would be willing to work on this if devs think it is a good idea but no one is working on it.

On a scale of 1 to "are you kidding me?", how much pain am I in for trying to make a threaded executor? I've found one whammy already with some sys.path mutation but I think I can work around that.

I guess I should say "multiple independent pipeline runs in threads"

Not a threaded executor for a single pipeline

Though I would like the latter eventually too 🙂

Question about conditionally executing visualizations when using Jupyter notebooks as Dagster solids: I have a pipeline that uses Jupyter notebooks (via

dagstermill

) as solids. Several of my notebook solids include plotting code that allows visualization of intermediate results. However, rendering these plots can be computationally expensive. I want to be able to execute the pipeline in two differents ways: (1) with execution of plotting code, when debugging/inspecting; (2) without execution of plotting code, when I just want the final results. The most obvious way to do this is to put the plotting code under a conditional that reads some parameter. To me, the most natural kind of parameter to use is what Dagster calls a [

mode

](https://docs.dagster.io/overview/modes-resources-presets/modes-resources). I prefer this to a solid configuration parameter because (a) my configuration parameters concern the values I'm computing-- whether I pre-compute visualizations or not is a kind of meta-parameter; (b) the visualization parameter should be shared across all solids. Really it is a kind of logging configuration, but I don't think that I can use dagster's built-in logging API because I am trying to control execution of notebook cells. So I'd like to use separate modes to toggle this visualization behavior. My problem is, from what I can tell from the

mode

documentation, modes simply configure resource/logger/storage/executor keys-- I can't figure out how to read the name of the mode itself from the context during execution. Is this possible? More generally, is there some more appropriate Dagster abstraction I should be using to control this behavior? Finally, it occurred to me while thinking through this that it would be nice if dagstermill itself were capable of varying its notebook cell execution dependent on cell metadata. That way, the visualizations I described above could be sequestered in specially tagged cells, and the notebook execution engine could conditionally execute them depending on the mode. I'm pretty sure that Papermill supports this functionality.

Hi Guys, may I have a question? I would like to invoke Dagit's API from another service. Actually I would like to start pipelines. execute_pipeline would be greate if I can give a remote endpoint, but I did not see this in the documentation. Maybe it is there. So my question is what would be the best way to achive remote execution. If I need a grapql client, can you segguest me something? I useally dont use GraphQL so my knownledge here is very limited (lets say I do not know anything about this ;) ) Thanks in advance

Link to @dwall's panel at Coalesce (happening in a few moments) https://www.getdbt.com/coalesce/agenda/when-metadata-becomes-data

Hi everyone - I have Dagit running inside a container; I can run one of my pipelines locally using a preset -- everything works for me. However, when a teammate, using the same container image and pipeline code, tries to run the same pipeline, he gets a "This pipeline run has been marked as failed from outside the execution context." and the run fails without any stack trace, etc. Any ideas what could be causing that? We're on 0.9.19

Hi everyone, I have a question about mode definitions and passing in config_schema to resource_defs: https://github.com/dagster-io/dagster/discussions/3385

Hi Guys, I have just started my 1st pipeline on K8s with celery executor. My question would be, how much CPU does Dagit need? I gave it 1000 milli core, but K8s throttling immediately start scheduling down. So i have started 5 Celery workera and my guess was Dagit only responsible to take pipeline executions, starts them, so put them into the celery broker queue (redis), and the rest, the calculations are disributed between celery workers. But it seems my celery workers are bored 🙂 and Dagit cannot have enough resource. Do you have CPU suggestion for Dagit? (this is in a stream processing environment, so we are hitting the GraphQL API)

hi all. Came across this issue regarding incremental re-execution, and was wondering how it is related to intermediates. Isn't the desired functionality described in the issue covered by them?

catherinewu just published a new version: 0.9.22. See https://github.com/dagster-io/dagster/releases/tag/0.9.22 for the release notes.

FYI: We’re currently releasing cherry-picked commits off of a

release

branch. Most of the commits are going into

master

and will be released with

0.10.0

(Jan 2021) 🍒

catherinewu just published a new version: 0.9.22.post0. See https://github.com/dagster-io/dagster/releases/tag/0.9.22.post0 for the release notes.

So finally following up on something I asked about like 2 months ago (sigh yak shaves): displaying an image (or more generally, arbitrary HTML) in the output log for a run.

Markdown metadata does allow images at least, but it's in a secondary modal

Is there any way to trick it into just inserting raw HTML into the metadata table itself?

Looks like no, from the code in MetadataEntry.tsx

Maybe a new HTML metadata type?

Also, asset keys, it seems like a 3 item list isn't a thing? It creates a broken link 🙂

It seems like only the first and last item in the list are actually used

Ahh no, I think the issue is asset keys with

/

in them

The link from the log isn't url-escaped

Hi Guys! Have you seen situation where Dagit is frozen after a few hundreds pipeline execution? Neither GraphQL API, nor Dagit Web cannot be opened / not responding. Is there a way to figure out what is happening?

Hi everyone! Apologies if this is a simple question but I've looked around the documentation, including around config_mapping and configured, but couldn't see anything for my use case. I'm trying to load a JSON that defines my data streams, iterate over those data stream definitions and then run parts of my pipeline based on those definitions. The issue I have is that because the iterative definitions are created in the pipeline not a solid I get a

DagsterInvalidDefinitionError

. I tried moving the loop into a composite_solid but received the same error. Is there a standard way to handle loops in Dagster?