Hi guys, I'm having some trouble setting up a schedule

Hi, I wanted to propose a change and was going to put up a small PR for your team to review, but don't have push access to the dagster repository. I went through the docs on Contributing (https://docs.dagster.io/latest/community/contributing) but it doesn't specify the process we should follow for PRs etc. What is the best way to do this / do I need to fork the repo etc?

Any plans on integrating dagster with slack (appart from the resource in dagster-slack)? Would be really helpful to receive messages in slack when a pipeline fails.

https://github.com/dagster-io/dagster/releases started getting pushed lots of releases via phabricator a day ago -- not sure if intentional but it makes reviewing recent semver releases a bit tough

is there anyway to connect two pipelines serially

?

Hello everyone!

Hello!

Hi, We want to understand what is the best practice for running solids with configurations files. Our simple use case is that we have a pipeline that we need to run on different files (each file with a different configuration). From this https://dagster.phacility.com/D2060 link and this PR #2049 we are not sure what is the preferred direction to take (the former one is more appealing to our use case). Any guidance would be appreciated.

Max Gasner just published a new version: 0.7.4.

Hi, In the Executing on Dask documentation you state that -

you may find that invoking Dask or Pyspark directly from within the body of a solid’s compute function is a better fit than the engine layer covered in this documentation.

Do you have an example to invoke Dask execution compute function from within a

solid

, we are running on a

cloudera

environment and need to run the solid process with dask-yarn. any advice would be appreciated.

hi! can anyone help me that, where the log files are stored? or do we need to configure to write log to file?

Hey team, I have an opinion based question. A couple of months ago (maybe a year) dagster recommended to deploy dagster using dask or airflow. However, in recent months A scheduler for dagster was released and recently better deployment options on k8. What were the reasons behind this shift and would you still recommend to deploy using airflow scheduler? And if anyone there tried and settled on one of both. What were your reasons? I’m currently exploring both (airflow and dagster scheduler). Personally I’m more inclined to the idea of airflow being a general use scheduler (cron on steroids) and have other data tools like dagster has native plugins mechanism with it.

Hi everyone ! I'm having trouble trying to pull the docker image dagster.io/dagster-docker-buildkite:py37-latest I got this error :

Error response from daemon: Get <https://dagster.io/v2/>: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Is anyone also having this issue ?

Question about project structure. What are people considering to be best practices for setting up a repository to handle pipelines pertaining to different lines of business?

how can i secure the dagit ui using username/password credentials

as I'm digging in a bit more: damn guys. Solid work 😉

^is that our communities first dagster pun 😆

Hi guys, I am trying to get "re-execution for solid" feature running on Dagit intro-tutorial, I am using the codes in: <dagster-official-repo>/examples/dagster_examples/intro_tutorial/ (latest master branch on github) I run:

dagit -f hello_cereal.py -n hello_cereal_pipeline

The dagit UI does not allow me to use re-execution, recommending me to use the "filesystem" storage mode. Where am I doing wrong ? I want the button "re-execution" to work. Dagit Page and intermediates.yaml file posted in thread of this post

Hello Again ! I've worked on my k8s integration of dagster. After launching a

helm install dagster helm/dagster

, my celery workers fail to start The result of a

kubectl logs pod-name

:

Traceback (most recent call last):
  File "/usr/local/bin/dagster-celery", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster_celery/cli.py", line 159, in worker_start_command
    pythonpath = get_config_dir(config_yaml)
  File "/usr/local/lib/python3.7/site-packages/dagster_celery/cli.py", line 69, in get_config_dir
    if 'broker' in validated_config:
TypeError: argument of type 'NoneType' is not iterable

if I run a kubectl exec on the pod, and look at the content of the celery-config.yaml, it looks like this

execution:
  celery:
      broker:  "<pyamqp://test>:test@dagster-rabbitmq:5672//"
      backend: 
      config_source:

So, it looks like the helm template is working as expected. Dos anyone has any advice about this ?

it seems like users aren't expected to supply explicit typing for the injected context argument -- what's the rationale there?

Put up a small PR # 2287 for dagster-aws don't know how your review process works lmk if there are any comments etc. Thanks!

Hi, how can I run

execute_solid

with a solid that has been aliased? I'm running into this error:

dagster.check.ParameterCheckError: Param "solid_def" is not a ISolidDefinition. Got <dagster.core.definitions.composition.CallableSolidNode object at 0x10f233c10> which is type <class 'dagster.core.definitions.composition.CallableSolidNode'>.

For example, I'd like to be able to do this:

some_alias = some_solid.alias('some_alias')
execute_solid(some_alias)

Hi again guys, I've got a question on pipelines today.

Hey Y'all I'm wondering if there is a way to point a

ScheduleDefinition

's

should_execute

filter to another pipeline having completed. I'm wondering if there is a way to set dependencies like Pipeline B executes at scheduled time if and only if the last run of Pipeline A successfully ran.

Hi everyone, I'm experimenting and learning about dagster for a few days now. So far I was just using it to run dags on my local machine, which was an exciting experience. Currently I'm wondering how I would integrate a new scheduler and execution engine. Say I have a running htcondor cluster and want to use it for sceduling and execution (it already supports dags on its own). Is there further documentation you could kindly point me out to? How do I start? The documentation mentions to say Hi and ask here 🙂

Hi everyone! I need to build a use case with Dagster to interact with EMR: spawning the cluster, adding steps to it, and shutting it down. I went trough the dagster_aws/emr code, but I could not find any example of usage of EMR. Can someone point me out to an example or the snippet of using the EmrRunJobFlowSolidDefinition ? thanks

Hi everyone, I'm trying to work through the local Dask example here: https://docs.dagster.io/latest/deploying/dask/ and I'm getting some errors about using an ephemeral DagsterInstance. When I write the config in the Dagit playground using

filesystem

, everything works great. When I use the Python API and pass the YAML config like

dagster pipeline execute -f dask_hello_world.py -n dask_pipeline -e dask_hello_world.yaml

, I get a

PIPELINE_INIT_FAILURE

error.