prha just published a new version: 0.7.11.post0.

After installing 0.7.11, trying to run

dagit -f file.py -n nodename

throws an error from (what seems to be) within dagster's config system. Anyone else seeing this?

File "c:\users\bsmith\appdata\local\continuum\anaconda3\lib\site-packages\dagster\config\field_utils.py", line 225, in is_potential_field
    potential_field
  File "c:\users\bsmith\appdata\local\continuum\anaconda3\lib\site-packages\dagster\config\field.py", line 117, in resolve_to_config_type
    desc=VALID_CONFIG_DESC,
dagster.core.errors.DagsterInvalidDefinitionError: You have passed an instance of DagsterType Path to the config system (Repr of type: <dagster.core.types.dagster_type._Path object at 0x0000019207D41F08>). The DagsterType and config schema systems are seperate. Valid config values are:

Hi all, hope you are well. I was wondering how would you go about restricting access for a dagit instance. Is there some available method for authentication?

Hi We are trying to understand how to define the configuration when running

execute_pipeline

. 1. We set the

DAGSTER_HOME

environment 2. When running

dagit

we can see that the logs are saved under the

storage

and under

history/runs

3. When running the

execute_pipeline

from within a python file the logs are not saved (under the environment_dict we have the configurations for the

solids

that we are running in the

pipeline

. What is the correct manner to configure

exectue_pipeline

? Update I guess we need to define the instance .

Hi, in my $DAGSTER_HOME/dagster.yaml I put exectuion_manager disabled, thinking that would prevent people from executing pipelines from that dagit instance. That appears to do the trick, playground is greyed out and re-execute button gives an error popup. The error is confusing though: AttributeError: Type "DauphinStartPipelineReexecutionDisabledError" not found in the Schema. So I'm looking for confirmation that putting above snippet is indeed a good way to prevent pipeline executions from that dagit instance (I only want to see previous execution logs in it)

has anyone gotten

context.pdb.set_trace()

to work with docker-compose? i added

stdin_open: true

and

tty: true

to my docker-compose service but it still doesn't let me interact with the terminal

is there a good way to route stdout to context.log? sometimes i wish i could log output to dagster's logger without having to go into the stdout tab, since i can't always access

context.log

in the libraries that i'm importing

the new docs look beautiful!! i love it

btw, it would be cool if there was a list of companies using dagster somewhere 😛

Sashank Thupukari just published a new version: 0.7.12.

the new assets tab is so nice. i completely love it. i think our GIS team will be happy

is there any way to debug the situation where some solids complete but the following solids never trigger?

Is there any way to remotely launch a pipeline run on a dagster instance with a http request?

i'm getting

Message: An error occurred (404) when calling the HeadObject operation: Not Found

when trying to view raw output, anyone seen this before?

Hi, Has there been any work done in order to compare between logs? e.g. Assuming we are running some pipeline and would like to compare between the results of the runs? or between specific steps between runs

Hi, After migrating from 0.7.3 to 0.7.12 with

dagster instance migrate

I get the following GraphQL errors on the`runs` tab in `dagit`:

Message: Cannot query field "pipelineSnapshotId" on type "Pipeline"
Path:
Locations: [{"line":38,"column":7}]

Message: Cannot query field "rootRunId" on type "PipelineRun". Did you mean "runId"?
Path:
Locations: [{"line":28,"column":3}]

Message: Cannot query field "parentRunId" on type "PipelineRun". 
Path:
Locations: [{"line":29,"column":3}]

To solve the current issue - I am going to start from a clean slate and keep the old db for reference if required

Has there been any effort to build out some sort of

awesome-dagster

resource to list community examples, resource plugins, etc. that other can leverage in their own work?

I'm looking at using a composite solid to wrap a bash_command_solid so that I can use config arguments to parameterize the command. I was also trying to use a results directory resource that I wrote to handle the output data of the command, but it looks like the composite_solid doesn't accept a resources option. I'm guessing the blessed approach would be to write a small solid that just generates the command data and output path and passes those to the bash solid?

Hello! I am new to dagster and trying to evaluate its feature set. Does it currently support the execution of arbitrary "solids" via docker containers? I stepped through the tutorials online and saw it mentioned within the airflow integration but did not find any other information

The runs tab in my dagit UI seems to take forever to load. Is there anything I can do to fix this? I’m running 0.7.6.

Related to my question above, but conceptually separate, if I have multiple outputs for a solid, how do I then attach those outputs to the proper parameters in the construction of the pipeline? Do I need to do that in a PipelineDefinition instance?

Hi! I was wondering how can you make dagster capture the logs of modules used inside solids that may not have direct access to the logging module. So for example if I have a module

import logging
logger = logging.get_logger(__name__)

class Processor:
    def process(self):
        logger.log("processing...")

I could see this log in the dagster execution logs when used in a solid

@solid
def main_process():
    processor = Processor()
    processor.process()

Currently I'd pass the context logger directly to the object, but I was wondering wether there could be a better approach.

Gitlab's Meltano project has announced a focus on becoming the open-source data extractor/loader alternative to Alooma, Fivetran, Stitch, Matillion, etc. The announcements on the refocus are here: • https://meltano.com/blog/2020/05/13/revisiting-the-meltano-strategy-a-return-to-our-roots/ • https://meltano.com/blog/2020/05/13/why-we-are-building-an-open-source-platform-for-elt-pipelines/ Meltano's approach reminds me of a narrower in scope and more opinionated Dagster, but packaged all together as a platform. They use Airflow for orchestration, and Stitch's Singer specification for taps and targets, DBT for transformations, and their own BI layer. I don't know how modular their extractor/loader portion is, but I'm happy to see the Singer specs getting more robust productionization options. If Dagster is interested in going farther upstream, Meltano would be an interesting project to follow.

I guess i'm missing something - How do I define a

dict

type within a solid configuration? I'm not able to understand how to define a general dict from the error : Passing

Dict

sets an error

DagsterInvalidDefintionError: Attempted to pass <dagster.core.types.python_dict.DagsterDictApi object> to a Field that expects a valid dagster type usable in config (e.g. Dict, Int, String et al).

The

Permissive

is too strict - I just need a general

dict

type Tried

Shape({})

> but still problematic

1. A Python primitive type that resolve to dagster config
   types: int, float, bool, str.

2. A dagster config type: Int, Float, Bool, String, StringSource, Path, Any,
   Array, Noneable, Selector, Shape, Permissive, etc.

3. A bare python dictionary, which is wrapped in Shape. Any
   values in the dictionary get resolved by the same rules, recursively.

4. A bare python list of length one which itself is config type.
   Becomes Array with list element as an argument.

Hi again! is there a way to specify schedule definitions via yaml?

hey, i'm looking to make a start on https://github.com/dagster-io/dagster/issues/2458 and basing it off the

dagster_aws.emr

subpackage. i've started with the types module, but it looks like the

dagster_aws.emr.types

module uses both regular python enums and dagster enums - why is that? have i missed something in the docs?

alangenfeld just published a new version: 0.7.13.

Hi, I wanted to ask you. If I start dagit with certain env variables (eg

$ VAR=val dagit ...

), how can I make these available for the scheduled jobs also, it seems they are not starting as the env vars are not set for the cron.