Quick q: how do I inject a custom

command

to the Job resource that is created from pipeline executions when using K8sRunLauncher? Thank you.

I'm getting an invariant error when I try to run from the failed solids (having changed some code with and restarted the pipeline container)

Hi all, How do i construct pipeline with yaml, with dependency graph that uses Nothing dependencies.

Is it possible to re-execute a solid after a quick code fix and keep it as part of a previous pipeline execution? Can imagine that'd be a nasty feature to develop but would be a game changer for pipeline development.

Hi team, I am so sad to say this but I am still getting troubles using Dagster in windows. My team installed Dagster in their machines (Windows) and randomly we are getting this error. It looks that Dagster has troubles running in windows machines because in the linux server we never faced the same issue.  

Exception: Timed out waiting for tail process to start

File "C:\Users\***\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\api.py", line 756, in pipeline_execution_iterator

for event in pipeline_context.executor.execute(pipeline_context, execution_plan):

File "C:\Users\***\Anaconda3\envs\borrar\lib\site-packages\dagster\core\executor\in_process.py", line 38, in execute

yield from iter(

File "C:\Users\***\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\api.py", line 835, in __iter__

yield from self.iterator(

File "C:\Users\***\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\plan\execute_plan.py", line 75, in inner_plan_execution_iterator

active_execution.verify_complete(pipeline_context, step.key)

File "C:\Users\***\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__

next(self.gen)

File "C:\Users\***\Anaconda3\envs\borrar\lib\site-packages\dagster\core\storage\compute_log_manager.py", line 56, in watch

yield

File "C:\Users\***\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__

next(self.gen)

File "C:\Users\***\Anaconda3\envs\borrar\lib\site-packages\dagster\core\storage\local_compute_log_manager.py", line 51, in _watch_logs

yield

File "C:\Users\***\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__

next(self.gen)

File "C:\Users\***\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\compute_logs.py", line 31, in mirror_stream_to_file

yield pids

File "C:\Users\***\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__

next(self.gen)

File "C:\Users\***\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\compute_logs.py", line 75, in tail_to_stream

yield pids

File "C:\Users\***\Anaconda3\envs\borrar\lib\contextlib.py", line 120, in __exit__

next(self.gen)

File "C:\Users\***\Anaconda3\envs\borrar\lib\site-packages\dagster\core\execution\compute_logs.py", line 104, in execute_windows_tail

raise Exception("Timed out waiting for tail process to start")

Hi team, also I am very confused about the tab partitions in Dagit. For example, following the error above, I tried to rerun several times the matrix partition but It doesn't show any error (red squares). If I rerun the matrix several times, some steps are successful and some are not. It is extremely random :(

Hello, we have recently hooked up some prometheus stats on our end and I was wondering what stats are included as part of

dagster-prometheus

?

Hi, can someone have a look at the change log for 0.12.6? It seems to be empty.

Hi team, I am looking to use the

dagster_snowflake

resource in my pipelines. What is the recommended way to set the password to the resource config? The password is available in the user-deployment as an env var. Should I just set the env var name in the resource config and make sure that the env var is set in the K8s job container?

We have an hourly scheduled pipeline run which checks for the existence of new files via SFTP then does some stuff with them. The "checks for existence of new files" idempotence seems like a good use case for the

run_key

of a Sensor, but let's suppose (I'm not sure this is actually the case...) we want to keep to a proper hourly schedule rather than specify a minimum interval of about an hour and allow it to drift. Is there a good way to accomplish this?

Dynamic fan out with s3 as io manager is giving me some problems? Just updating my Dagit and dagster version to check the problem still exists.

Hi, I'm trying to use dagster for a data orchestration pipeline and started analyzing it recently, where we expect to dynamically create and manage them from a custom UI, my question is, does dagster support REST endpoints for creation of solids and pipelines? Thinking of creating a fastapi based BE which will be responsible for creation, management and executions of dagster. Can we store them in some DB or s3 in yml format? read about it somewhere in docs but not clear for me.

Would it ever be reasonable to be able to search through logs via dagit? Or shall I hook up elastic search or similar / stop being lazy and just query the db?

Hello! I’m trying to trigger my Dagster pipeline (which does some basic Pandas data processing and outputs some parquet files) to run when a file is uploaded to Azure blob storage. I’m having some trouble getting it deployed using Azure Functions. Has anyone done something similar? I’m also open to implementing the trigger through Dagster rather through Azure Functions if anyone has experience with that! Many thanks!

Hi all. I currently have a two pipelines: pipelineA which processes a single file and pipelineB which reads a whole bucket in s3 and yield asset materializations for each file it finds. I would like to use the pipelineB to read the whole bucket in s3 and launch an instance of pipelineA for each file it finds. From what I can see, sensors would accomplish this task. However, I would like to run it on a ad-hoc basis and not have to have the sensor constantly running in the background. Any suggestions for how I could organize this? Thanks!

I love the new core APIs (https://dagster.io/blog/core-apis-moving-towards-1-0). The decoupling of graphs and jobs, and feeding jobs directly to schedules+sensors, is great. I also like the new names (graph, op, job). The sense of “job” aligns with how the term is used in Geewax’s recent API Design Patterns book, so that’s nice as well. Great job.

How do I specify pod annotations for the Kubernetes job pods?

Getting an

Exception: Timed out waiting for tail process to start

running a pipeline in Windows 10, dagster version 0.12.6:

Hi All, does Dagit/Dagster has any user signon /role based access control ?

created a issue for a potential UI bug: https://github.com/dagster-io/dagster/issues/4530

I'm a beginner with dagster and logging. I want capture all dagster logging output (compute logs, execution logs etc) to Datadog. Without affecting the default dagster logs that go to dagit and dagster daemon. While we learn dagster our goal is to have dagster log output searchable (in Datadog) alongside logs from our non-dagster data applications. What is the simplest way to do that? I looked at dagster Loggers but these seem to be about pipeline level logging. I'm initially looking for something coarser than that - with no or minimal configuration required in dagster. Could I collect logs at machine/container level using Datadog agent, logstash etc? I can see some log settings in dagster.yaml:

event_log_storage

,

compute_logs

. But I don't know if these represent 'all dagster logging output', or how to configure each to log to multiple locations: a place convenient for dagit/dagster-daemon (postgres) AND a place that a logging agent can read them.

hey guys, what is this process?

/export/apps/python/3.7/bin/python3.7 -E -s -c from multiprocessing.semaphore_tracker import main;main(19)

I want to use solid memoization but don't want to have to structure my pipelines with a solid per file (or graphs / ops etc.). I was planning on generating hashes from the string representation of functions. Not sure how awkward it will be to access the solid definition from the @solid() wrapper. Anyone tried something like this?

Hi, guys, I'm having this pipeline that processes different N classes of input, and I'm trying limit concurrency for pair

(pipeline, input-class-N)

, I don't mind if runs for

(pipeline, input-class-I)

and

(pipeline, input-class-J)

runs in parallel, but there should be only 1

(pipeline, input-class-I)

at any time And I can't really figure out how to do that, I can do that with

QueuedRunCoordinator

specifying

tagConcurrencyLimits

for each input class, but it looks ugly. Wouldn't it be much better, if I could define something like

type: QueuedRunCoordinator
      config:
        queuedRunCoordinator:
          tagConcurrencyLimits:
            - key: non-concurrent
              limit: 1

and then for each run define tags like

tags={"non-concurrent": "pipeline-input-class-I"}

? And it looks like it wouldn't be that hard to do in _TagConcurrencyLimitsCounter that will change the api unfortunately, so maybe it would be better to be able to use custom RunCoordinatorDaemon? What do you think, is there any plans for that? Does any of that sounds good enough so I could make pull request?

Hi Team, I need a bit of help for the deployment in AWS using dagster projects, unfortunately couldn't found in documentation So a bit of context with the simple solids and pipelines using repo.py is working perfectly file. But the problem starts to occur in aws when I change the structure of solid and pipelines in a new directory project. So the objective is not to use repo.py for the trigger of pipelines (here is the example which shows I am refereeing too ). This line 65 of docker-compose.yaml file uses this command '' dagster api grpc -h 0.0.0.0 -p 4000 -f repo.py " and the same command goes to our AWS infrastructure for the trigger of pipelines. Instead what I am looking for is to utilise the workspace.yaml file (where i can add multiple python packages ). So does anyone think can be command be used like this '*dagster api grpc -h 0.0.0.0 -p 4000 -w workspace.yaml*' ? (presently there is no existing '-w' parameter) If not then another Idea is to use Module instead of 'repo.py' in the main directory. Dagit works really well with the module ' *dagit -m _project-01_*'. but can this be possible with the dagster ? so the command would become like this '*dagster api grpc -h 0.0.0.0 -p 4000 -m _project-01'_* (presently it throws an error that project-01 don't exists )

I get this error when I try to re-run from a solid that is used as input to two parallel streams (not sure if that's the cause)

How do I stream Dagster logs to frontend, not Dagit rather my own custom frontend, any help will be appreciated

Good Morning, I am writing some sphinx documentation and wanted to use Dagster in intersphinx, I was wondering if dagster had any URL similar to this one(https://numpy.org/doc/stable/objects.inv) to allow for dagster docs to be referenced/hyperlinked automatically? Thank you and appreciate the great work!

How do you guys typically deploy dagster + dagit in larger companies? Do you have a central deployment everybody submits PRs to? Do you deploy an instance for each team? For each project?

everything I’ve found on deployment so far has a (hardcoded) coupling from the dagit/dagster deployment back to the repositories that are supposed to be running - which means that a central “infra team” would have to redeploy each time another team would like to add a project. Is that the case?