dagster #announcements

schrockn

02/11/2019, 4:01 PM

@max i liked that paper enough that tweetstormed about it just now https://twitter.com/schrockn/status/1094989230760112128

max

02/12/2019, 11:32 PM

Copy code

We do use Airflow as our orchestration tool.
What is the best practice to handle dependencies among models when we have both DBT and Airflow? Is it good to use DBT inbuilt DAG or use Airflow to handle that dependency?

schrockn

02/12/2019, 11:33 PM

where is that front

schrockn

02/12/2019, 11:33 PM

*from

max

02/12/2019, 11:33 PM

dbt slack

max

02/12/2019, 11:33 PM

https://www.eventbrite.com/e/dbt-community-meetup-data-council-pregame-at-envoy-tickets-56208302546

schrockn

02/12/2019, 11:34 PM

nice

schrockn

02/12/2019, 11:34 PM

registered

max

02/13/2019, 7:22 PM

(more an index of the difficulty of the current tooling i think than anything else) https://www.locallyoptimistic.com/post/analytics-engineer/

max

02/21/2019, 7:15 PM

1000 commits to dagster:master! 👯‍♀️

🙌 1

Taylor

03/03/2019, 5:02 PM

@schrockn @max You know what'd be a rad feature of the framework? A built in way to diff a dag. Here's the problem I'm running into. As the number of collaborators increase, there are sometimes unintentional dag structural changes introduced. These are often difficult to find unless everyone is constantly reloading

dagit

, and able to see 1 solid change in a giant dag (not an optimal human task). I used to use this rad "perceptual diff" tool https://github.com/bslatkin/dpxdt for front end work to catch visual regressions and I'm thinking something like this, but not running via pixel by pixel diffs - but by executing against the internal representation of the dag itself. Then you could imagine dag snapshots being built into an org's pipeline CI:

Dagbot reports these dag changes: ✅ to confirm intention, ❌ to indicate this was unintentional.

👍 1

max

03/04/2019, 10:22 PM

http://science.sciencemag.org/content/sci/suppl/2017/12/27/358.6370.1530.DC1/aap8062-Brynjolfsson-SM.pdf

Taylor

03/14/2019, 9:01 PM

👋 Hi! @max @schrockn I submitted a patch this morning and a new release

v0.2.8.post1

and I'm baffled how what I changed broke this:

Copy code

(venv) 001: [develop x?] $ pip list | grep dag
dagit               0.2.8.post0 
dagster             0.2.8.post1 
(venv) 001: [develop x?] $ python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:07:29) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dagster
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "repos/project/venv/lib/python3.6/site-packages/dagster/__init__.py", line 1, in <module>
    from dagster.core.execution import (
  File "repos/project/venv/lib/python3.6/site-packages/dagster/core/execution.py", line 28, in <module>
    from .definitions import (
  File "repos/project/venv/lib/python3.6/site-packages/dagster/core/definitions/__init__.py", line 1, in <module>
    from .context import PipelineContextDefinition
  File "repos/project/venv/lib/python3.6/site-packages/dagster/core/definitions/context.py", line 5, in <module>
    from dagster.core.types import Field, Dict, String
  File "repos/project/venv/lib/python3.6/site-packages/dagster/core/types/__init__.py", line 2, in <module>
    from .config_schema import (
  File "repos/project/venv/lib/python3.6/site-packages/dagster/core/types/config_schema.py", line 2, in <module>
    from dagster.utils import single_item
ImportError: cannot import name 'single_item'
>>>

I cannot replicate this on the new

0.2.8.post1

branch.

Taylor

03/14/2019, 9:01 PM

This is in a fresh clean virtualenv

max

03/14/2019, 9:19 PM

are you getting that error only on 0.2.8.post1?

Taylor

03/14/2019, 9:26 PM

Correct. Not on .post0

Taylor

03/14/2019, 9:26 PM

which makes little sense since the delta is so small

max

03/14/2019, 9:48 PM

i can confirm the issue, i wonder if it's something to do with the release process

max

03/14/2019, 9:48 PM

did you run the release @Taylor?

Taylor

03/14/2019, 9:49 PM

@schrockn ran the release. I'm super curious how you confirmed the issue

max

03/14/2019, 9:49 PM

i installed 0.2.8.post1 into a new virtualenv and ran

from dagster.utils import single_item

Taylor

03/14/2019, 9:50 PM

From a pip install or local install

max

03/14/2019, 9:50 PM

pip

Taylor

03/14/2019, 9:50 PM

ah, there's the diff - I couldn't repo locally

max

03/14/2019, 9:50 PM

so i bet that the build uploaded to pypi is broken somehow

Taylor

03/14/2019, 9:50 PM

makes sense.

Taylor

03/14/2019, 9:50 PM

how do we test that?

schrockn

03/14/2019, 9:51 PM

back

schrockn

03/14/2019, 9:51 PM

@max i just ran the publish.sh script that existed in that repo

schrockn

03/14/2019, 9:51 PM

did you try installing using the git style requirements