Hi. Firstly, awesome work on this project. I was wondering, is there a way to define a Docker image within a solid? This was I could have a pipeline that may use multiple docker images.

Can you please guide me how to run coverage along with the "Pipeline Unit Test" example

Hi all 🙂 Is anyone here using Dagster with Databricks? I’ve seen a dagster-databricks module, but it’s not very well documented, so I’d be keen to hear about your setup

Hi guys, is there any documentation on

PartitionScheduleDefinition

?

Hello guys, I am trying to create a Cron Scheduler, and meeting Error:

dagster.core.errors.ScheduleExecutionError: Error occurred during the execution of run_config_fn for schedule <name>

More Info in the description!

We're seeing an error in our scheduler post 0.8 -> 0.9 migration

dagster.check.CheckError: Failure condition: Attempted to deserialize class "FileInDirectoryCodePointer" which is not in the serdes whitelist.

Any thoughts?

Hey,

k8s-celery-worker:0.9.3

docker image hasn't been pushed to https://hub.docker.com/r/dagster/k8s-celery-worker/tags. is this registry a supported source for images? or should we build our own images? thanks

Hi everyone, firstly just want to say what an awesome product you have created here. I have a question regarding the celery-k8s-job-executor. Is it possible to set k8s pod limits and requests per pipeline config when using this executor? If so how would one do it? Would be great to be able to do this so we can specify for example gpu pod resources.

Hey, I'm trying to set up Dagster with 

CeleryK8sRunLauncher

 and 

celery-k8s

 executor. Using s3 as storage. To give jobs access to S3 buckets I used

dagster-k8s/config

tag and annotate the pods with the right role. When I'm running the pipeline:

An exception was thrown during execution that is likely a framework error, rather than an error in user code.
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden

The only pod which doens't have the role annotation is

CeleryK8sRunLauncher.

I'm wondering is this error coming from that and if yes how can I annotate it?

I've got sqlalchemy autoflush enabled in the db model code that my solids run, and sometimes a pipeline run fails due to an integrity constraint failure that happened in a totally different pipeline. Seems the same celery worker executed both pipelines and reused the db session between the runs without first cleaning up after the first one caused an exception. This problem with celery+sqlalchemy is discussed here http://www.prschmid.com/2013/04/using-sqlalchemy-with-celery-tasks.html, and before adopting dagster we were extending celery.Task and calling

session.remove()

in its after_return method, just like that article recommends. What's the cleanest way to do this in dagster?

Sashank Thupukari just published a new version: 0.9.4.

Hi! Could you please help me to figure out how to test pipeline if it uses some external resources? Are there proper ways to so? Or the only way is to set up an environment (DB, etc) and write special test mode for the pipeline? I tried to use execute_pipeline and solid_selection. But I cannot figure out how to pass inputs in solids in the middle. code in the thread

Are those resources defined within the pipeline?

Hello. I installed dagster and dagit in a conda env and am trying to go through the basic tutorial for dagster. Was able to create and run hello_dagster.py with

dagster pipeline execute -f hello_dagster.py

but can't get http://localhost:3000 to open now (on any browser). Just get a "Firefox can’t establish a connection to the server at localhost:3000."

<!channel> Hi all! We’re starting a new recurring community meeting for Dagster, which will take place on the second Tuesday of every month. At the start of each meeting, the Dagster team will provide updates on new features and changes, as well as discuss upcoming features and future plans. Depending on interest, we will then have a community member give a lightning talk or demo. (If your organization is interested in sharing how you use Dagster, please reach out to us!) Finally, there will be room for questions and having general discussions. Our first community meeting is on Tuesday, September 8th at 9 am Pacific Time (UTC-7). Please DM me your email or fill out this form to be added to the calendar invite: https://forms.gle/K2dni4iPq48GnUEq6. If you aren’t able to make it, we’ll share a recording afterward. Here’s a preview of the agenda for the first meeting: https://docs.google.com/document/d/17wvfmyOLpADLnkoSnG9kmyDuyuvlz4suCNPKk2L2xHs/edit?usp=sharing

Hi Team! I have a question on concurrent runs. Does pipeline manage concurrent runs on it own? Can we control how many parallel threads for my pipeline?

Hello guys - I’ve just got aware about Dagster and I’m evaluating it as a possible candidate to orchestrate a big ETL system… I’m going through the tutorial; however I got stuck in a problem. I’ve logged a question to the StackOverflow. I appreciate if anyone can help addressing it: Dagster: Multiple and Conditional Outputs (Type check failed for step output xxx PySparkDataFrame)

Hi - QQ: Does Dagster has support to run the DAG’s solids inside separated Kubernetes containers? I can see in the documentation how to run Dagster in the K8s, but I didn’t find information about spreading the solids among containers (not sure if it does make sense according to Dagster architecture). For example, do we have (or will have soon) something analog to Airflow K8s operators, or executors?

Hi dagster team! We are using dagster in our production pipeline. Our pipelines have different yaml files depending on the environment (e.g. staging, production). Currently we are facing a problem when changing our yaml files for the run config. It already happened some times that our yaml files aren’t valid, but we only discovered it when the pipeline was executed in production. Is there any possibility to check for each pipeline if the yaml files contain valid configurations for the pipeline before the pipeline is executed?

Hi - Can you please explain this error for me or point me to a previous response?

dagster.check.ParameterCheckError: Param "msg" is not a str. Got {'id': 'sofijlaksdjfow', 'fields': {'id': 1, 'name': 'tester'}, 'createdTime': '2020-08-21T21:37:44.000Z'} which is type <class 'dict'>.

Here is the solid:

@solid 
def get_lead_parameters(context):
    lst = []
    responses = json.loads(requests.get('')['data']
    for res in responses:
        lst.append(res)
   return lst

And the above error is printed. This is the first solid and the next one just takes the list of dict . Thanks

Hi guys, I've just reconfigured my instance to use postgre for storage, using env var references to supply the credentials in dagster.yaml. Everything went very smoothly, but now my schedules are failing silently. I'm 99% sure the issue that these new variables dont exist for the cron user, but I'm unsure on how to configure this properly and could use some guidance.

prha just published a new version: 0.9.5.

Hey everyone, here is the Zoom link for our community meeting in 15 minutes (9am PST): https://us02web.zoom.us/j/88939083535 Agenda: https://docs.google.com/document/d/17wvfmyOLpADLnkoSnG9kmyDuyuvlz4suCNPKk2L2xHs

had to check out a little early to hop to another meeting, but excellent community meeting, y'all! Looking forward to more of these in the future 👍

Thanks for the community meetup! You can reach out to me here as well if you have any question about our Dagster work.

Thanks for organizing that! It was great to see how Prezi is using Dagster and looking forward to more of these in the future! dagsir

great presentations everyone! looking forward to all the new features and to more meetups in future 👏

Is it possible to point a dagster repository to a local folder?