the recycle icon next to the repository name on the left

I'm curious how Dagit gets the repo data now though 🙂

Oh I think I see, there's a grpc subprocess started automatically

Hi, I know that Dagster can pass data between solid directly(to take benefits such as data validation, type checking, etc), It's mean that all of the data will be stored in RAM right? then If we need to deal with a large dataset(larger than RAM). How to deal with it?

I'm probably missing something obvious, but I'm trying to figure out how to programmatically kick off execution of a pipeline from python without having to wait for the results. My use case is something like a streamlit app as a frontend where people can submit input data to be processed in an hours-long process. For short processes, I'm able to use

execute_pipeline()

, but I don't want to have Streamlit spinning for hours while it waits for completion. Do I have to go straight to Dask or Celery to be able to do this or can I accomplish something like this as a test case with just 1 processor/thread?

@Ben Lindsay Are you using Dagit?

Something somewhere needs to be running, or at least there isn't a fully async version of the executor system, but usually it would be a subprocess of dagit 🙂

(or a k8s job, or a subprocess of a grpc server)

@Noah K Yeah I have Dagit running, but I don't want my users to have to learn how to use Dagit. I'd like to have a simpler frontend to expose just what they need, and hook into the existing dagster instance. I figured since you can run multiple pipelines simulaneously from Dagit (actually I guess I'm assuming that as well), there should be a way to hook into the instance to kick off a pipeline externally or something like that, but maybe that's not true?

You can write a little CLI that calls the mutation, heres an example

Johann Miller just published a new version: 0.9.17.

Can

dagster api grpc

load a workspace.yaml or just one file/module at a time?

Looks like the latter from the --help output

I think this is probably a @sandy question but can I use

configured()

on an executor? I should probably just try it and find out 🙂

Am I correct in my reading of the code that StepLaunchers are a bit magical and that if you one (and exactly one) in the resources for a solid it will just get used automatically?

I think that's what this code in the execution context system does at least

And all the pyspark examples seem to work like that 🙂

Hmm, I can't seem to find the step inputs in the step_context object

Guessing I need to use the intermediate_storage and solid.input_definitions to find them?

https://github.com/dagster-io/dagster/blob/9f220f20f3e440123216168cd913fa637cf94819/python_modules/dagster/dagster/core/execution/plan/execute_step.py#L473 has the relevant code but it's private

I might just make a solid factory instead

It seems like the use case for StepLaunchers is more like writing the solid code itself in a DSL of some kind.

@Noah K, Namespaces Threads are one honking great idea -- let’s do more of those! 😉

Heh, next on my list is actually a thread based executor

hi guys, i'm a total noob here, managed to run dagit within docker that has volume

./app:/opt/dagster/app

which contains the pipeline

hello_cereal.py

&

workspace.yaml

. I made a syntax mistake in

PurePath.join()

instead of

PurePath.joinpath()

inside

hello_cereal

, but when I fixed it, dagit UI doesn't seem to hot reload the newest script even after page reload.Do I have to create a new pipeline?

Hi guys, we have a concern on scheduling: our dagster scheduler isn’t running — we have done the ff: • Verified that schedules are in crontab • Job runs properly when we manually execute the command listed in crontab • Cron logs are empty (including

mail

) • Jobs don’t show up in SQLite Dagster version: • 0.9.16 Does anyone else have ideas on how we can debug this problem? Thanks a lot!

This is nothing vital but i'm a bit confused by this, and it makes me curious. I use docker-compose to spin up an instance of postgresql container to handle every configurable storage in

dagster.yaml

, but even though I declared

environment:
      - POSTGRES_DB=dagster
      - POSTGRES_USER=dagster
      - POSTGRES_PASSWORD=dagster

I don't see any table named

dagster

inside postgres or in the schema. What is the function of

POSTGRES_DB

if not for pointing which table to use by

dagster_postgres.run_storage

, etc?

None of the tables are called dagster