I don't really know what stringsource and intsource mean

I think it's for things that can come from the environment instead?

Hello, I like a lot of Dagsters concepts, but there are hardly any companies referenced using it. Is it currently ready for production use, or still in beta phase?

Also, what is the revenue model behind Dagster? Will there e.g. be paid addons or paid support options? Does Elementetl have enough funding to continue development of Dagster in the long term?

Hi all, do you know by any chance, if the env-notation is not supported for pyspark? Somehow it doesn’t pick up my envs properly. I tried several times. If I replace the env with the real values it works. Or am I doing something wrong? 🤔

dagpurr  Hi all! Dagster community meeting is starting now at 9 am Pacific Time (UTC-7). Please check in our Agenda to join our zoom meeting to learn about 0.10.0 new features!

asset sensor 🤯

nice works team! im selling really hard to my team to switch to Dagster however the MySQL support is the deal-breaker. hopefully we can support it soon.

such a great meeting - so many 🤯 moments. Excellent work Elementl team 👏

dagpurr  Hi all! Thank you for your time and participation in today's Dagster community meeting. We had many great questions and conversations. Here's a feedback form if you didn't see the link I posted. Thank you for filling it out in advance!

Hiya! When would the features demoed in yesterdays community meet be available in github? Any specific branch to check out?

Thank you! 🙂

🙂

Hey Experts! Any best practices for the directory/file hierarchy and naming for repos, pipelines, composite solids and solids?

I'm way out off the normal path but my layout so far is

<reponame>-pipelines

for the pipelines and solid shells and then

<reponame>-<solidname>

for each solid daemon but most people don't have those 🙂

Hey all, has anyone got a code example of an SFTP > S3 intermediate storage solution please?

Is it already possible to deploy and run custom command line tools on a Dagster Kubernetes deployment? E.g. Oracle sqlplus

recording for this meeting please

We just tweeted about the highlights of the meeting! https://twitter.com/dagsterio/status/1349410333908078593?s=20

The full recording is here:

Hi everyone. I am new to Dagster and was wondering if anyone had advice on how set up dependencies between solids where data does not need to be shared. Essentially I am looking for a way to have one solid start after the completion of another (or multiple). Thanks!

It seems like the new auto-reload, once a gRPC connection is marked as Failed, it will never reconnect on its own?

Hello! I'm testing out Dagster with PySpark. I've got a few solids with DagsterPySparkDataFrame as inputs/outputs. My config is declaring storagefilesystemconfig:base_dir to a directory on my server, and the first run of the pipeline works flawlessly - looks awesome! I tried to Re-Run a segment of the DAG but am getting a error raised from Pyspark about the intermediate parquet file already existing. I'm assuming there's some way to tell the system to always re-run intermediates with mode="overwrite"?

Hey! I want to generate a pipeline using Yaml, is it required to generate the python file or is there a dagit API which could be directly plugged into?

prha just published a new version: 0.10.0. See https://github.com/dagster-io/dagster/releases/tag/0.10.0 for the release notes.

🎊

Are there any guarantees on certain things remaining free after dagster goes commercial? E.g. will hosting your own dagster instance with the features that are available now always be free?

Is there an estimate on when 0.10.x will show up in conda-forge?

I’ve noticed on occasion that the when running DAGs in the playground the Waterfall animation either doesn’t run or will run continuously even after the job has either succeeded or failed. Usually I end up waiting for the “Running” pill to change to succeeded or failed and then click over to the Run Log in the UI to inspect the run. Ultimately it’s not a huge deal because the jobs still run but the first time it happened I definitely just thought something was wrong with my job. Just curious if this is a common issue or if maybe I’ve done something in my jobs that’s causing some event to not emit to the client interface.