https://dagster.io/ logo
Join the conversationJoin Slack
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Powered by Linen
announcements
  • m

    matas

    05/09/2020, 9:01 AM
    btw, who is your hosting\dns provider? From Moscow I can only access https://docs.dagster.io/ via VPN I've checked, it is not connected with any kind of federal restrictions
    y
    s
    • 3
    • 3
  • u

    user

    05/09/2020, 6:18 PM
    prha just published a new version: 0.7.11.post0.
  • b

    Ben Smith

    05/10/2020, 2:40 PM
    After installing 0.7.11, trying to run
    dagit -f file.py -n nodename
    throws an error from (what seems to be) within dagster's config system. Anyone else seeing this?
    File "c:\users\bsmith\appdata\local\continuum\anaconda3\lib\site-packages\dagster\config\field_utils.py", line 225, in is_potential_field
        potential_field
      File "c:\users\bsmith\appdata\local\continuum\anaconda3\lib\site-packages\dagster\config\field.py", line 117, in resolve_to_config_type
        desc=VALID_CONFIG_DESC,
    dagster.core.errors.DagsterInvalidDefinitionError: You have passed an instance of DagsterType Path to the config system (Repr of type: <dagster.core.types.dagster_type._Path object at 0x0000019207D41F08>). The DagsterType and config schema systems are seperate. Valid config values are:
    s
    • 2
    • 12
  • c

    Cris

    05/11/2020, 2:36 AM
    Hi all, hope you are well. I was wondering how would you go about restricting access for a dagit instance. Is there some available method for authentication?
    a
    • 2
    • 2
  • s

    sephi

    05/11/2020, 7:03 AM
    Hi We are trying to understand how to define the configuration when running
    execute_pipeline
    . 1. We set the
    DAGSTER_HOME
    environment 2. When running
    dagit
    we can see that the logs are saved under the
    storage
    and under
    history/runs
    3. When running the
    execute_pipeline
    from within a python file the logs are not saved (under the environment_dict we have the configurations for the
    solids
    that we are running in the
    pipeline
    . What is the correct manner to configure
    exectue_pipeline
    ? Update I guess we need to define the instance .
    a
    • 2
    • 3
  • e

    ezechiel syx

    05/11/2020, 10:42 AM
    Hi, in my $DAGSTER_HOME/dagster.yaml I put exectuion_manager disabled, thinking that would prevent people from executing pipelines from that dagit instance. That appears to do the trick, playground is greyed out and re-execute button gives an error popup. The error is confusing though: AttributeError: Type "DauphinStartPipelineReexecutionDisabledError" not found in the Schema. So I'm looking for confirmation that putting above snippet is indeed a good way to prevent pipeline executions from that dagit instance (I only want to see previous execution logs in it)
    Untitled
    a
    y
    • 3
    • 2
  • c

    Chris Roth

    05/11/2020, 7:09 PM
    has anyone gotten
    context.pdb.set_trace()
    to work with docker-compose? i added
    stdin_open: true
    and
    tty: true
    to my docker-compose service but it still doesn't let me interact with the terminal
    m
    • 2
    • 17
  • c

    Chris Roth

    05/11/2020, 7:48 PM
    is there a good way to route stdout to context.log? sometimes i wish i could log output to dagster's logger without having to go into the stdout tab, since i can't always access
    context.log
    in the libraries that i'm importing
    p
    • 2
    • 5
  • c

    Chris Roth

    05/11/2020, 9:17 PM
    the new docs look beautiful!! i love it
    :dagsir: 5
    ❤️ 2
  • c

    Chris Roth

    05/11/2020, 9:17 PM
    btw, it would be cool if there was a list of companies using dagster somewhere 😛
    👍 1
  • u

    user

    05/11/2020, 11:01 PM
    Sashank Thupukari just published a new version: 0.7.12.
    :blob-clap: 3
    🔥 2
  • s

    sashank

    05/11/2020, 11:04 PM
    0_7_12_Release_Notes
    😛artydagster: 5
    ❤️ 3
    😛artywizard: 3
  • c

    Chris Roth

    05/12/2020, 1:33 AM
    the new assets tab is so nice. i completely love it. i think our GIS team will be happy
    ➕ 1
    🙏 2
  • c

    Chris Roth

    05/12/2020, 2:02 AM
    is there any way to debug the situation where some solids complete but the following solids never trigger?
    a
    • 2
    • 13
  • a

    Auster Cid

    05/12/2020, 8:27 PM
    Is there any way to remotely launch a pipeline run on a dagster instance with a http request?
    s
    t
    w
    • 4
    • 12
  • c

    Chris Roth

    05/12/2020, 9:28 PM
    i'm getting
    Message: An error occurred (404) when calling the HeadObject operation: Not Found
    when trying to view raw output, anyone seen this before?
    m
    • 2
    • 1
  • s

    sephi

    05/13/2020, 8:55 AM
    Hi, Has there been any work done in order to compare between logs? e.g. Assuming we are running some pipeline and would like to compare between the results of the runs? or between specific steps between runs
    a
    p
    • 3
    • 23
  • s

    sephi

    05/13/2020, 11:55 AM
    Hi, After migrating from 0.7.3 to 0.7.12 with
    dagster instance migrate
    I get the following GraphQL errors on the`runs` tab in `dagit`:
    Message: Cannot query field "pipelineSnapshotId" on type "Pipeline"
    Path:
    Locations: [{"line":38,"column":7}]
    
    Message: Cannot query field "rootRunId" on type "PipelineRun". Did you mean "runId"?
    Path:
    Locations: [{"line":28,"column":3}]
    
    Message: Cannot query field "parentRunId" on type "PipelineRun". 
    Path:
    Locations: [{"line":29,"column":3}]
    To solve the current issue - I am going to start from a clean slate and keep the old db for reference if required
    s
    a
    • 3
    • 2
  • t

    Tobias Macey

    05/13/2020, 2:39 PM
    Has there been any effort to build out some sort of
    awesome-dagster
    resource to list community examples, resource plugins, etc. that other can leverage in their own work?
    s
    s
    • 3
    • 3
  • t

    Tobias Macey

    05/13/2020, 7:26 PM
    I'm looking at using a composite solid to wrap a bash_command_solid so that I can use config arguments to parameterize the command. I was also trying to use a results directory resource that I wrote to handle the output data of the command, but it looks like the composite_solid doesn't accept a resources option. I'm guessing the blessed approach would be to write a small solid that just generates the command data and output path and passes those to the bash solid?
    m
    a
    b
    • 4
    • 19
  • p

    pranavsingh

    05/13/2020, 8:02 PM
    Hello! I am new to dagster and trying to evaluate its feature set. Does it currently support the execution of arbitrary "solids" via docker containers? I stepped through the tutorials online and saw it mentioned within the airflow integration but did not find any other information
    a
    • 2
    • 3
  • t

    Tim Chan

    05/13/2020, 8:50 PM
    The runs tab in my dagit UI seems to take forever to load. Is there anything I can do to fix this? I’m running 0.7.6.
    a
    • 2
    • 2
  • t

    Tobias Macey

    05/13/2020, 8:56 PM
    Related to my question above, but conceptually separate, if I have multiple outputs for a solid, how do I then attach those outputs to the proper parameters in the construction of the pipeline? Do I need to do that in a PipelineDefinition instance?
    s
    • 2
    • 2
  • c

    Cris

    05/13/2020, 10:15 PM
    Hi! I was wondering how can you make dagster capture the logs of modules used inside solids that may not have direct access to the logging module. So for example if I have a module
    import logging
    logger = logging.get_logger(__name__)
    
    class Processor:
        def process(self):
            logger.log("processing...")
    I could see this log in the dagster execution logs when used in a solid
    @solid
    def main_process():
        processor = Processor()
        processor.process()
    Currently I'd pass the context logger directly to the object, but I was wondering wether there could be a better approach.
    p
    m
    • 3
    • 13
  • n

    Nam D. Nguyen

    05/14/2020, 3:46 AM
    Gitlab's Meltano project has announced a focus on becoming the open-source data extractor/loader alternative to Alooma, Fivetran, Stitch, Matillion, etc. The announcements on the refocus are here: • https://meltano.com/blog/2020/05/13/revisiting-the-meltano-strategy-a-return-to-our-roots/ • https://meltano.com/blog/2020/05/13/why-we-are-building-an-open-source-platform-for-elt-pipelines/ Meltano's approach reminds me of a narrower in scope and more opinionated Dagster, but packaged all together as a platform. They use Airflow for orchestration, and Stitch's Singer specification for taps and targets, DBT for transformations, and their own BI layer. I don't know how modular their extractor/loader portion is, but I'm happy to see the Singer specs getting more robust productionization options. If Dagster is interested in going farther upstream, Meltano would be an interesting project to follow.
    s
    s
    • 3
    • 4
  • s

    sephi

    05/14/2020, 8:54 AM
    I guess i'm missing something - How do I define a
    dict
    type within a solid configuration? I'm not able to understand how to define a general dict from the error : Passing
    Dict
    sets an error
    DagsterInvalidDefintionError: Attempted to pass <dagster.core.types.python_dict.DagsterDictApi object> to a Field that expects a valid dagster type usable in config (e.g. Dict, Int, String et al).
    The
    Permissive
    is too strict - I just need a general
    dict
    type Tried
    Shape({})
    > but still problematic
    1. A Python primitive type that resolve to dagster config
       types: int, float, bool, str.
    
    2. A dagster config type: Int, Float, Bool, String, StringSource, Path, Any,
       Array, Noneable, Selector, Shape, Permissive, etc.
    
    3. A bare python dictionary, which is wrapped in Shape. Any
       values in the dictionary get resolved by the same rules, recursively.
    
    4. A bare python list of length one which itself is config type.
       Becomes Array with list element as an argument.
    s
    • 2
    • 4
  • c

    Cris

    05/14/2020, 4:06 PM
    Hi again! is there a way to specify schedule definitions via yaml?
    a
    • 2
    • 3
  • b

    Ben Sully

    05/14/2020, 4:36 PM
    hey, i'm looking to make a start on https://github.com/dagster-io/dagster/issues/2458 and basing it off the
    dagster_aws.emr
    subpackage. i've started with the types module, but it looks like the
    dagster_aws.emr.types
    module uses both regular python enums and dagster enums - why is that? have i missed something in the docs?
    s
    • 2
    • 8
  • u

    user

    05/14/2020, 11:15 PM
    alangenfeld just published a new version: 0.7.13.
    🔥 1
  • a

    alex

    05/14/2020, 11:20 PM
    0_7_13
    🔥 5
    😛artywizard: 1
    :congadagster: 4
Powered by Linen
Title
a

alex

05/14/2020, 11:20 PM
0_7_13
🔥 5
😛artywizard: 1
:congadagster: 4
View count: 1