https://dagster.io/ logo
Join the conversationJoin Slack
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Powered by Linen
announcements
  • c

    Chris Roth

    04/14/2020, 8:45 PM
    also, how do i get the raw log output to work? i'm currently using sqs/celery and seeing nothing when i view raw logs
    p
    • 2
    • 11
  • m

    Muthu

    04/14/2020, 9:39 PM
    #general getting this error
    ModuleNotFoundError: No module named 'dagster.serdes'
  • m

    Muthu

    04/14/2020, 9:40 PM
    dagster --version dagster, version 0.7.4
  • m

    Muthu

    04/14/2020, 9:45 PM
    DAGSTER_HOME=/opt/dc/workflow_engine/app /Users/muthu/miniconda3/envs/workflow/bin/dagit -f <http://dcpipes.py|dcpipes.py> -n define_repo -h 0.0.0.0 -p 3000
    Traceback (most recent call last):
      File "/Users/muthu/miniconda3/envs/workflow/lib/python3.6/site-packages/dagster/core/serdes/__init__.py", line 185, in rehydrate
        module = importlib.import_module(self.module_name)
      File "/Users/muthu/miniconda3/envs/workflow/lib/python3.6/importlib/__init__.py", line 126, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 994, in _gcd_import
      File "<frozen importlib._bootstrap>", line 971, in _find_and_load
      File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "<frozen importlib._bootstrap>", line 994, in _gcd_import
      File "<frozen importlib._bootstrap>", line 971, in _find_and_load
      File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 678, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/Users/muthu/miniconda3/envs/workflow/lib/python3.6/site-packages/dagster_postgres/__init__.py", line 1, in <module>
        from .event_log import PostgresEventLogStorage
      File "/Users/muthu/miniconda3/envs/workflow/lib/python3.6/site-packages/dagster_postgres/event_log/__init__.py", line 1, in <module>
        from .event_log import PostgresEventLogStorage
      File "/Users/muthu/miniconda3/envs/workflow/lib/python3.6/site-packages/dagster_postgres/event_log/event_<http://log.py|log.py>", line 17, in <module>
        from dagster.serdes import (
    ModuleNotFoundError: No module named 'dagster.serdes'
    
    During handling of the above exception, another exception occurred:
    d
    m
    a
    • 4
    • 6
  • c

    Chris Roth

    04/14/2020, 9:47 PM
    Has anyone been able to get celery and SQS to work? It is creating a new queue for every run for me. I've tried changing backend between ampq and rpc but same thing happens either way. It seems like this is causing it to run on my AWS workers no matter what I do instead of my local celery worker
    m
    • 2
    • 5
  • m

    Muthu

    04/14/2020, 10:21 PM
    hi… thanks for your help… i’ve updated to 0.7.6… and connected event logs to Postgres… after triggering the pipeline, its not update the logs on the dagit immediately… it needs browser refresh
    m
    • 2
    • 3
  • j

    John Mav

    04/15/2020, 4:09 PM
    I’ve been following the examples here: https://dagster.readthedocs.io/en/stable/sections/deploying/local.html#dagit-in-docker to get Dagit running within a docker container. I have a simple pipeline that I am using to test the cron service that gets setup and everything seems like it’s running fine. However when I turn on the pipeline schedule from the schedules tab in the UI the pipeline never runs? The cron service is running and when I list the jobs from the crontab I do see the specific test job e.g.
    root:/opt/dagster/app# crontab -l
    
    * * * * * /var/shared/dagster/schedules/scripts/Repo.test_run.sh # dagster-schedule: Repo.test_run
    I’m wondering what I am missing here so that turning “on” the schedule actually runs the job?
    s
    • 2
    • 6
  • c

    Chris Roth

    04/16/2020, 5:52 PM
    i noticed that the runs page in dagit takes a long time to load when i have a lot of runs - anyone else encountering this?
    a
    • 2
    • 12
  • c

    Chris Roth

    04/16/2020, 6:00 PM
    also, feature request - it would be nice if i could click on the tags in the runs page to add the tag to the filter search bar
    a
    p
    • 3
    • 4
  • c

    Chris Roth

    04/16/2020, 6:00 PM
    having a hard time copying and pasting tags, search bar doesn't like pasting
  • c

    Chris Roth

    04/16/2020, 6:03 PM
    also! it would be cool if i could see which celery worker each solid is running on
    m
    n
    • 3
    • 6
  • c

    Chris Roth

    04/16/2020, 7:17 PM
    I'm noticing something mysterious. I have 4 celery workers set up with Redis and I'm running a pipeline with about 6 serial solids in it. I wrote the solids before setting up celery so they read and write a json file to and from disk instead of passing as an intermediate. But somehow the downstream solids are succeeding even on celery. How is this possible? It seems kind of like all of the solids in a pipeline are always getting run on the same worker
    n
    • 2
    • 10
  • c

    Chris Roth

    04/16/2020, 8:18 PM
    haha so many questions for you guys today! it's deploy day. i'm also setting a bunch of "This pipeline run has been marked as failed from outside the execution context"
    a
    • 2
    • 35
  • c

    Chris Roth

    04/16/2020, 10:03 PM
    every time i deploy dagit, the schedule gets set to "off". can this be setting get stored in postgres?
    a
    s
    • 3
    • 9
  • u

    user

    04/17/2020, 5:44 AM
    Max Gasner just published a new version: 0.7.7.
  • m

    max

    04/17/2020, 5:49 AM
    0_7_7_Release_Notes
    🤘 3
    😛artydagster: 3
    :congadagster: 2
    😛artywizard: 2
    ❤️ 1
    🔥 7
  • t

    Tobias Macey

    04/17/2020, 5:36 PM
    Working with types in Dagster, is there a defined type for the Dagster context object, or would that just be a regular
    dict
    type?
    a
    • 2
    • 3
  • c

    Chris Roth

    04/17/2020, 11:19 PM
    is there a way to pass config options from dagster-celery to celery? i'm getting
    You're running the worker with superuser privileges
    in docker. celery accepts
    --uid
    option but not
    docker-celery
    m
    • 2
    • 5
  • c

    Chris Roth

    04/18/2020, 12:31 AM
    getting
    dagster.core.errors.DagsterLaunchFailedError: Host <http://localhost:3000> failed sanity check. It is not a dagit server.
    after upgrading to 0.7.7
    s
    m
    p
    • 4
    • 63
  • b

    Binh Pham

    04/20/2020, 5:32 PM
    is there any built-in watermarking in dagster? to do incremental runs
    a
    y
    • 3
    • 10
  • c

    Chris Roth

    04/21/2020, 3:16 PM
    my scheduler isn't running 😞 it is scheduled for 3am but i don't see anything in the logs
  • c

    Chris Roth

    04/21/2020, 3:17 PM
    it is enabled, and the entrypoint script is:
    #!/bin/sh
    
    # See: <https://unix.stackexchange.com/a/453053> - fixes inflated hard link count
    touch /etc/crontab /etc/cron.*/*
    
    service cron start
    
    # Migrate dagster Postgres database
    dagster instance migrate
    
    # Add all schedules
    dagster schedule up
    
    # Restart previously running schedules
    dagster schedule restart --restart-all-running 
    
    dagit -h 0.0.0.0 -p 3000
  • c

    Chris Roth

    04/21/2020, 3:18 PM
    p
    s
    +2
    • 5
    • 109
  • t

    Tobias Macey

    04/21/2020, 4:34 PM
    Working on the first stage of a pipeline and I believe I recall that Dagster doesn't currently have a means for running a fan-out/fan-in topology. The context being that I have a solid that generates a list of values and I want to execute the subsequent stages in sequence for each of those outputs. e.g. generate_list_of_strings fans out to do_something_with_string, do_something_else_with_string, etc. so that I have 5 solids running in parallel for item 1 in that list, and then repeat for each of the other strings.
    m
    • 2
    • 6
  • c

    Chris Roth

    04/21/2020, 6:13 PM
    i'm having an issue where one of my solids is getting stuck (not sure why, but i assume it's a hangup with my solid code and not a dagster bug), but it would be cool if there was a way to force terminate it so i could start it over
    m
    c
    • 3
    • 5
  • c

    Chris Roth

    04/21/2020, 7:16 PM
    another question: sometimes i want to run solids in a certain order, but i don't want the input from the previous ones. is that possible, or is the only way to force them to run in series to have them use the input from the previous one?
    p
    m
    • 3
    • 5
  • r

    Res Dev

    04/21/2020, 8:21 PM
    Anyone implemented reusable email alert plugin whenever a pipeline fails?
    a
    s
    m
    • 4
    • 12
  • r

    Res Dev

    04/21/2020, 8:37 PM
    I have SMTP env variables available, could write code to send email in case of exception in solid. But that means repeating that everywhere in every solid.
    m
    • 2
    • 1
  • r

    Res Dev

    04/21/2020, 8:38 PM
    I’m curious if we can detect “Failed” in any solid context in a pipeline and call a on_failed_callback with context error message.
  • r

    Res Dev

    04/21/2020, 8:38 PM
    Similar to airflow
Powered by Linen
Title
r

Res Dev

04/21/2020, 8:38 PM
Similar to airflow
View count: 1