set the channel topic: Next community meeting Feb 9th 2021! Sign up here: https://bit.ly/3lYAvhf. Subscribe to our Youtube channel http://bit.ly/2XMyY3z for past events videos

I am currently running 0.9.22 on a raspberry with the cron scheduler dockerized and that works just fine. I am currently trying to migrate to 0.10.. Did I understand that correctly that I need a second container for the scheduler daemon - as done in the docker deployment example? Or can I still run everything in a single container? Also: Good morning

I'm currently leaning heavily on AWS Lambda and Step Functions for pipeline compute. As expected, orchestration is a nightmare. Are there plans to revive the lambda execution engine (https://github.com/dagster-io/dagster/issues/455)? In the mean time, are there any docs on rolling your own executors with the executor API (https://docs.dagster.io/_apidocs/internals#executors)? Is executing via Lambda a dead end for now?

Maybe I missed something in the docs... is there a recommended way in dealing with secrets on a dev laptop? I'd probably use StringSource configuration to get secrets on Kubernetes, but how can I fill the StringSource on a local machine without commiting anything to git? .env file with dotenv might be a way to do that, but that doesn't seem to be the standard scenario in dagster as I can't find any references to it.

We would like to stand up a bare dagster server into which, over time, we will deploy dagster-ready packages with their own repositories, workspace.yaml etc. Initially, when it is stood up, the only packages deployed to the server are the dagster core libs and dagit. Dagit will be started and point to a workspace.yaml. The workspace.yaml is expected to grow over time as we add python_package entries to it. Initially however, we need to point to something - a package, a module that will allow dagit to start. I notice there is a dagit_tests.toy.bar_repo that seems to do the trick. Can this be relied on to always be present or is there another better alternative?

Hello I've upgraded to 0.10.0. I noticed that more often than not, after clicking on the green "Launch Execution" button in dagit, the "Checking config" animation doesn't occur. I get no feedback at all, then about a few seconds later the gantt chart tab does open and the pipeline does run fine. My setup is local with on-prem Postgres backend. Using Chrome version 87 on Windows 10 OS. Has anyone else experienced this also?

This message was deleted.

Hey! How would I add configuration to a sensor, like with a solid please?

Hello everyone! We just pushed a blog post detailing the major changes in 0.10.0. I think there will be interesting content even for those who attended the community meeting. https://dagster.io/blog/dagster-0-10-0-the-edge-of-glory

The dagster account tweet: https://twitter.com/dagsterio/status/1351590844743094279

Add my tweetstorm summarizing the blog post: https://twitter.com/schrockn/status/1351604432199290880

You are encouraged to share and retweet widely 🙏🏻🙏🏻🙏🏻

Hi everyone, I've problems with Docker Deployment. I deployed docker example and the repo is loaded correctly in Dagit UI, but when i launch it, the daemon runs de pipeline but throw an error. Dagster.core.errors.DagsterImportError: Encountered ImportError:

No module named 'redd_motodb'

while importing module repo from file /opt/dagster/pipelines/repo.py. Local modules were resolved using the working directory

/opt/dagster/pipelines

. If another working directory should be used, please explicitly specify the appropriate path using the

-d

or

--working-directory

for CLI based targets or the

working_directory

configuration option for

python_file

-based workspace.yaml targets.

I'm loading repo.py with CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "repo.py"] like the example

Sorry, solved.

Any idea about this? SchedulerDaemon - ERROR - Scheduler failed for tbportal_temp_sync_schedule : dagster.check.CheckError: Invariant failed. Description: Reconstructed repository location should have exactly one repository

Hi, I have just implemented the scheduler in the latest version, no problem with the daily_scheduler however when I run the cron based one it doesn't seem to pass the mode onto the execution. I get the following error

2021-01-21 15:41:31 - SchedulerDaemon - ERROR - Failed to fetch execution plan for success_email_schedule: dagster.core.errors.DagsterInvalidConfigError: Error in config for pipeline success_email_pipeline
    Error 1: Missing required config entry "solids" at the root.

Stack Trace:
  File "c:\python\dagster\venv\lib\site-packages\dagster\grpc\impl.py", line 328, in get_external_execution_plan_snapshot
    create_execution_plan(
  File "c:\python\dagster\venv\lib\site-packages\dagster\core\execution\api.py", line 668, in create_execution_plan
    environment_config = EnvironmentConfig.build(pipeline_def, run_config, mode=mode)
  File "c:\python\dagster\venv\lib\site-packages\dagster\core\system_config\objects.py", line 128, in build
    raise DagsterInvalidConfigError(

My code for the daily_schedule (which works) is:

@daily_schedule(
    pipeline_name="summary_email_pipeline",
    start_date=datetime.datetime(2021, 1, 21),
    execution_timezone="Europe/London",
    mode="prod",
    execution_time=datetime.time(21, 00),
)
def summary_email_schedule(date):
    return

However the one which doesn't is...

@schedule(
    cron_schedule="*/1 * * * *", pipeline_name="success_email_pipeline", execution_timezone="Europe/London",
    mode="prod",
)
def success_email_schedule(context):
    return

Any ideas? Have been trying different things, can't seem to crack it.

I’m looking for a way to publish dataset documentation for the Assets materialised by my Dagster Pipelines. Any recommendations? What you use, and what do you like/dislike about it?

Rex Ledesma just published a new version: 0.10.1. See https://github.com/dagster-io/dagster/releases/tag/0.10.1 for the release notes.

Curious, in dagjt, is there an option where for the selected pipeline, only the schedules related to that pipeline are listed? Currently, it lists all schedules regardless of chosen pipeline. I guess that might be tough since currently there isnt a way to unselect a pipeline, and thus have the option to also see all schedules.

What is the reason that the docs recommend using a separate postgres server instead of the one deployed with the helm chart?

Also, is there any documentation on how to use DagsterDaemonSchedulers max_catchup_runs parameter? Would be great if a default could be defined in the Helm Chart and be overriden for individual schedules... maybe its already in the code?

Because the included chart, IIRC, has no HA provisions

Same for the included RabbitMQ

And I'm guessing the Dagster team doesn't really want to get into recommending (or worse, maintaining, since helm/charts is gone now) specific stuff for that

ok if the Chart for postgres in Dagster isn't maintained it makes sense

If you want something to run inside K8s, Zalando's postgres operator was very good the last time I used it

Though I'll always recommend hosted SQL products like RDS and CloudSQL as 100% worth the money 😄

Hi all. I've been evaluating some tools to use in a simple machine learning pipeline that will be run from a docker container. High overview of the flow is: pre-process the data -> stitch the labels -> train (produce model) and/or predict The data scientist in theam had already created code that uses MLFlow, we just need an ease of use pipeline to help with the flow. The tools for flow we are now at a cross roads are Dagster and Kedro. I have done tutorials on both. My view from doing the tutorials that Dagster was clear and obvious what it is happening, and how to use it. Kedro is opinionated on the project setup (but this can be good so their is a form of best practice in their), but it was a bit difficult to make out how everything gets stitched up to form a pipeline. However Kedro offers an advantage, it has an MLFlow plugin that seems to make it something to lean more towards. Is there anything of similar nature with Dagster or examples of how to setup ML pipelines?   Another question I have regarding Dagster: if a solid fails in the middle of the pipeline and I fix whatever the issue was, would I need to rerun the entire pipeline or just that solid and the pipeline continues from where it left off?   Sidenote: Metaflow was also a contender but the tutorials don't really give an idea what's going on, and from my reading it is really tied down to AWS kind infrastructures.

Hello, I’m trying to implement a web app that submits a pipeline to be executed to the QueuedRunCoordinator that runs in the background without hanging until the pipeline is complete. Is there a way to implement this with just the Python API or is the best way to just wrap a call to dagster.cli.pipeline.execute_launch_command? The pipelines themselves will probably be fairly lightweight but some solids will be submitting and waiting for a longer running job to finish on AWS Batch.