In defining a schedule, is there a way to specify which preset to use for a given pipeline?

I would like to execute the same pipeline code using 2 different presets to load separate configurations for different target sources on a daily schedule for each.

I've got a doozy of a question: running

dagit -f dagstermill_nb_pipeline.py

runs beautifully; it imports my other .py files and no problems all the pipelines from the repository run. I take this same file and use it with my k8s deployment with grpc and the dagit pod always fails complaining that my .py files weren't found as modules... I've attached the k8s log associated to the failed dagit pod

Can anyone help me to implement dagster with Azure databricks and ADLS gen2?

Rex Ledesma just published a new version: 0.9.10.post0.

Rex Ledesma just published a new version: 0.9.11.

.

Hey,

ModuleNotFoundError: No module named 'celery.backends.amqp'

I'm getting this error on all of our pipelines. Full error in thread. (let me know if this should go to

Discussions

)

hi all, i'm trying to add dagster to an open-source project i'm working on, got stuck, and thought i'd come here for some guidance. basic issue is i need to call an API (Socrata) with each of a list of keys and merge the results...

Hi, what are the options for executing solids in a pipeline in parallel? Is it possible to declare sets of solids which should be executed in parallel?

So, it seems that in one of the recent releases, if a solid doesn't explicitly yield an output it's counted as an error now? I was just running through a pipeline that had worked (maybe circa ~7.x?) where it implicitly returns None, and just saw a failure about not returning a non-optional output "result". So it seems I need to explicitly yield a Nothing result?

Hello everyone! Just wanted to ask here if I missed something before starting a discussion on github. I want to have a reusable solid that concats datasets. And I want to use it in different pipelines with arbitrary input number. How this can be achieved? Of course I can make concat-2 solid and stack them, but it looks like a huge graph complexity overkill and brings a lot of overheads on serializing stuff back and forth. Is there a way currently to implement it or if not, does it looks like a good feature for the rest?

Hi dagster team,

Is there a way to add role based access to the dagit UI. Currently we are migrating a production batch from CONTROL-M to dagster. We would like to restrict access through user_id and password and assign a read_only role to certain group of users.

There's nothing built in, so what I ended up using was Caddy with the auth plugin. I'm also evaluating Pomerium as an option for handling access and authorization.

Hello All, I am using Dask executor specifying local filesystem storage

"storage": {"filesystem": {"config": {"base_dir": '/dagster'}}}

, in the run_config. However I get surprising error

Undefined field "storage" at the root. Expected:

. Wonder if the storage has been overwritten somewhere? I also set mode as follows local_mode =

ModeDefinition(

name='local',

resource_defs={

's3': s3_resource,

'tempfile': tempfile_resource,

'file_cache': fs_file_cache,

'file_manager': local_file_manager,

},

executor_defs=default_executors + [dask_executor],

intermediate_storage_defs=s3_plus_default_intermediate_storage_defs,

system_storage_defs=s3_plus_default_storage_defs,

logger_defs={'local_logger': json_file_logger}

Dagster version 0.9.9. Thanks for suggestions in advance!

Hello, I am running into a problem with failure hooks and I don't know how to solve it. Basically, when I run a test code with success_hook and failure_hook, the success one works fine, it posts into Slack when it runs, but the failure_hook doesn't, the code raises the error but it does nothing. What could it be?

Hi! How can I get all the PipelineRunsErrors' traceback so I can post them into Slack?

Does anyone have a good example of using the FileManager system in a way that handles local vs. S3 based on a mode definition?

Hello again! How can I return a list of a pipeline's failed runs?

Hello All! Quick question. I set up

dagster.yml

with

PostgresRunStorage

as

run_storage:

module: dagster_postgres.run_storage

class: PostgresRunStorage

config:

postgres_db:

username:

env: DAGSTER_PG_USERNAME

password:

env: DAGSTER_PG_PASSWORD

hostname:

env: DAGSTER_PG_HOST

db_name:

env: DAGSTER_PG_DB

port: 5432

and similar for events using

PostgresEventLogStorage

. I then execute pipeline with dask executor - it runs successfully but gives quite a few errors

qlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123145510014976 and this is thread id 123145494249472.

I am curious why

SQLite

is mentioned at all since I use

Postgres

for run storage? Thanks for clarifications!

Hi all, I am curious if dagster is participating in hacktoberfest 🤩

Can I get some help configuring a custom celery executor?

gibsondan just published a new version: 0.9.12.

First guest post on our blog from the inimitable @dwall! https://twitter.com/davidjwallace/status/1311752683905191936

Hi 🙂 I'm trying to dabble with Dagster to see if it fits my ML team's use cases, we are currently on Airflow but Dagster seems really promising and way more ergonomic to use, especially for rapid pipeline development! I'm facing an issue trying to run

dagit

locally connected to a remote Kubernetes cluster with Dagster installed. I'm trying to use the

K8sRunLauncher.

Dagit starts fine and I can see my pipeline, but when i trigger it and check the pod logs I see this error:

dagster.check.ParameterCheckError: Param "pipeline_run" is not a PipelineRun. Got None which is type <class 'NoneType'>.

the args of the pod look like this:

api
      execute_run_with_structured_logs
      {"__class__": "ExecuteRunArgs", "instance_ref": null, "pipeline_origin": {"__class__": "PipelinePythonOrigin", "pipeline_name": "pipeline", "repository_origin": {"__class__": "RepositoryPythonOrigin", "code_pointer": {"__class__": "FileCodePointer", "fn_name": "pipeline", "python_file": "/Users/[redacted]/Projects/ml-platform/dsdk/pipeline.py", "working_directory": null}, "executable_path": "/Users/[redacted]/Projects/ml-platform/dsdk/.venv/bin/python"}}, "pipeline_run_id": "9a037a94-c7cc-4fd0-a737-ab65d46b5ab4"}

The local paths look suspicious but are they the cause for the error? This is my launcher config but I'm not sure if it's relevant

run_launcher:
    module: dagster_k8s.launcher
    class: K8sRunLauncher
    config:
        dagster_home: 
            env: DAGSTER_HOME
        instance_config_map: dagster-instance
        service_account_name: dagster
        postgres_password_secret: dagster-postgresql-secret
        job_image: [REDACTED].<http://dkr.ecr.us-east-1.amazonaws.com/dsdk:dagster|dkr.ecr.us-east-1.amazonaws.com/dsdk:dagster>
        job_namespace: datascience
        load_incluster_config: false
        image_pull_policy: Always

can anyone help me? Thanks in advance! And thanks again for this great project 🙂

(hosted dagit works fine, but i'd like a more interactive way of working on the pipelines without needing to publish new containers so i'm trying to use local dagit)

Max Gasner just published a new version: 0.9.13.