dagster #announcements

David Lacalle Castillo

09/09/2020, 9:02 AM

Is kerberos and ldap supported? What about service roles?

David Lacalle Castillo

09/09/2020, 12:29 PM

I have deployed dagster in Docker, I have created create a workspace.yml with 1 repo at beggining. Etrypoint is executing: DAGSTER_HOME=/opt/dagster/dagster_home dagit -h 0.0.0.0 -p 3000

David Lacalle Castillo

09/09/2020, 12:30 PM

I would like to be able to add dags dinamically. I created a new dag and repository inside docker container and updated workspace.yml to point to the new repository. However, dagit website is not showing the new repository. Is it possible to add repositoreis dinamically? Or just pipelines can be added dinamically?

David Lacalle Castillo

09/09/2020, 3:31 PM

Scheduler is not working: I have configured dagster.yml:

scheduler:

module: dagster_cron.cron_scheduler

class: SystemCronScheduler

Entrypoint executes this:

touch /etc/crontab /etc/cron.*/*

service cron start

# Add all schedules defined by the user

dagster schedule up

Error:

dagster_web  |

dagster_web  | Error: A scheduler must be configured to run schedule commands.

dagster_web  | You can configure a scheduler on your instance using dagster.yaml.

dagster_web  | For more information, see:

dagster_web  |

dagster_web  | <https://docs.dagster.io/deploying/instance/#scheduler>

daniel

09/09/2020, 3:37 PM

That sounds like it's not finding your dagster.yaml - just confirming, DAGSTER_HOME is pointing at the right place on dagster_web?

daniel

09/09/2020, 3:39 PM

Does your entrypoint.sh have "export DAGSTER_HOME=/opt/dagster/dagster_home" (or whatever the right directory is?)

David Lacalle Castillo

09/09/2020, 3:41 PM

The error was the mounting point, sorry

David Lacalle Castillo

09/09/2020, 3:49 PM

@daniel I have seen that when you dinamically add a new schedule in the interface you new to press a button for realoding schedules. Maybe automatically loading is a good idea.

David Lacalle Castillo

09/09/2020, 3:50 PM

The other thing that I would like to have in new version is the ability to load all files that are added into a folder.

David Lacalle Castillo

09/09/2020, 3:51 PM

The fact that you need to restart dagit for seing a new workspace is strange

David Lacalle Castillo

09/09/2020, 3:51 PM

I would like to load all the definitions from a folder of files, so you can dinamically add files with repositories and pipelines withouth the need of restarting the dagit service

David Lacalle Castillo

09/09/2020, 3:52 PM

What do you think about it?

David Lacalle Castillo

09/09/2020, 4:09 PM

What do you think on allowing pipeline versioning?

sashank

09/09/2020, 4:37 PM

Here’s the recording of yesterday’s community meeting:

https://www.youtube.com/watch?v=6uM1IDFjobs&feature=youtu.be▾

Slides for all the presentations are linked in the description

ladydaga 5

👍 1

dagpurr 3

docster 4

dwall

09/09/2020, 4:59 PM

ladydaga

dwall

09/09/2020, 4:59 PM

amazing

schrockn

09/09/2020, 5:00 PM

credit to @yuhan for that gem 🙂

👩‍🎨 1

😆 2

🎨 1

➕ 1

David Lacalle Castillo

09/09/2020, 5:13 PM

@prha and I are discussing of an approach that does not need to restart dagit and modify workspace.yml for adding new python files or packages. I tried pointing workspace.yml to a file and in that file importing others as bellows: from hello_dagster import * However it fails:

David Lacalle Castillo

09/09/2020, 5:13 PM

Copy code

dagster_web  | Stack Trace:
dagster_web  |   File "/usr/local/lib/python3.7/site-packages/dagster/cli/api.py", line 150, in list_repositories_commanddagster_web  |     python_file, module_name, working_directory, attribute
dagster_web  |   File "/usr/local/lib/python3.7/site-packages/dagster/grpc/utils.py", line 20, in get_loadable_targets   
dagster_web  |     else loadable_targets_from_python_file(python_file, working_directory)
dagster_web  |   File "/usr/local/lib/python3.7/site-packages/dagster/cli/workspace/autodiscovery.py", line 11, in loadable_targets_from_python_file
dagster_web  |     loaded_module = load_python_file(python_file, working_directory)
dagster_web  |   File "/usr/local/lib/python3.7/site-packages/dagster/core/code_pointer.py", line 181, in load_python_file
dagster_web  |     raise error
dagster_web  |   File "/usr/local/lib/python3.7/site-packages/dagster/core/code_pointer.py", line 129, in load_python_file
dagster_web  |     module = import_module_from_path(module_name, python_file)
dagster_web  |   File "/usr/local/lib/python3.7/site-packages/dagster/seven/__init__.py", line 116, in import_module_from_path
dagster_web  |     spec.loader.exec_module(module)
dagster_web  |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
dagster_web  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
dagster_web  |   File "/opt/dagster/dagster_home/hub.py", line 1, in <module>
dagster_web  |     from hello_dagster import *
dagster_web  |

Pete Fein

09/09/2020, 6:54 PM

anyone used a single machine + user code in venvs approach for production deploys? I have a client with limited devops capability, trying to keep it simple for them

sephi

09/10/2020, 11:15 AM

Enjoyed listening to @Tobias Macey and @schrockn on podcast._init_

dagsir 5

Sachit Shivam

09/10/2020, 1:13 PM

Hi guys Is it possible to use Celery workers and RabbitMQ running on different EC2 instances? I have dagster running on a particular EC2 instance and would like to use different EC2 instances to run my Celery workers and RabbitMQ Is that possible, and if yes, how do I configure that? Also, do I need to set a separate Celery broker up?

schrockn

09/10/2020, 3:43 PM

We have a real homepage now! https://dagster.io/ Thanks to @sashank, @yuhan, and @Ben Gotow for pulling this together ladydagaladydagaladydaga

🚀 3

docster 6

💯 4

🤩 2

ladydaga 14

sashank

09/10/2020, 5:26 PM

We’re excited to announce a new integration between Dagster and Great Expectations. GE enables Dagster users to build data quality checks directly into their pipelines, making it easier to catch data issues early. More on our blog: https://dagster.io/blog/great-expectations-for-dagster …

dagsir 8

David Lacalle Castillo

09/10/2020, 5:27 PM

Hi guys, I need to execute dask code inside a solid. This dask code connects to external cluster. Is it possible? @daniel @prha

David Lacalle Castillo

09/10/2020, 5:45 PM

If i use dask executor and each dask node has dask installed and connects to other cluster?

shin

09/10/2020, 9:45 PM

Hi folks, Dagster novice here - I was using Dagit yaml config playground to launch pipelines and now would like to translate this into graphql that can be run on graphql playground. From error message I see that Dagit yaml config playground will send graphql to execute the pipeline. Is there a way I can get a copy of graphql query I've sent using the UI, so I can use and modify in graphql playground (instead of translating configs to graphql myself)?

user

09/10/2020, 11:40 PM

catherinewu just published a new version: 0.9.6.

cat

09/10/2020, 11:55 PM

0_9_6_Changelog

🎉 3

Kevin

09/11/2020, 4:55 PM

general question: is there any appetite/future plans to be able to have solids defined within .ipynb files? so one .ipynb could contain multiple solids rather than the entire notebook being defined as a single solid?