in particular, my source files are dated CSVs. i want to have a pipeline that can essentially take a date as an argument and then do all the operations required for a given day, figuring out file names along the way. i'd like to then be able to use that to run, say, an entire year at once

what's the best way to do that? do i need to define the date as a resource? should i pass through the date as an argument to each solid using the API?

i notice there's something involving backfill and partitions in the latest git checkins, but not in the latest pypi package, or in the docs

also... is there no way to have dagit load in a config yml file?

Happy new year folks! Just wanted to share this little snippet to use Python enums in solid configs and hear your thoughts on it. Is this something you already have a dedicated function for (couldn’t find anything in the docs) or something that conceptually doesn’t make sense?

def as_dagster_enum(e: enum.EnumMeta):
    return Enum(e.__name__, [EnumValue(v.name, python_value=v) for v in e])

You can then use it in your solid config like so

class Strategy(enum.Enum):
    SOMETHING = enum.auto()
    SOMETHING_ELSE = enum.auto()


@solid(
    config={"strategy": Field(as_dagster_enum(Strategy)),
    …
)
…

Hi all, I’m setting up dagster to manage a data pipeline to write to google cloud storage and google bigquery. Locally it runs but when I deploy it to a simple server I get the following error, any clue what could be causing this?

Hello guys, it seems that the documentation for configuring instances is broken

I'm interested in testing a use case where I have a fairly simple dag that I have to run in an environment that is not mine and using dagit to inspect the results

thanks!

I'll take a look at the provided link, thank you max

Any reason why a dag execution wont exit when a step fails? I'm running with dagster pipeline execute

some best practice questions for y'all - A very common pattern for us is to execute a SQL query on Snowflake, fetch the results, and return the result set as a pandas DataFrame (most of the time saving an artifact of the dataset in cloud storage as well). Is there already an established pattern in Dagster for this? Is it recommended to decompose this kind of pattern into multiple solids (one for running the query, one for constructing the DataFrame from the result set, etc) or rather just house this kind of operation in a single solid? second question: we need to use the result set mentioned above ^ and feed it into multiple downstream solids that can all be run in parallel. These solids are largely identical and only differ by a small piece of config. Is there a way to programmatically generate these solids with different config values and have them all be able to be run in parallel?

Do I always need a repository.yaml for starting a dagit? It isnt possible to just point it to a remote storage?

Any target day for a new official release past 0.6.6? I'm waiting on the new pyspark testing feature that allows me to hold the session open to assert on output, but it's only in nightly builds presently, and when I attempt to do nightly builds, dagit starts to give issues, so hoping for a 0.6.7 release soon to mvoe forward with a testable pipeline. 🤞

Max Gasner just published a new version: 0.6.7.

Hey all. We have a lot of exciting stuff coming. We’d like to encourage as many folks as possible to develop off of master. It’s actually quite stable. I always demo off master (🤞🤞🤞) and it goes just fine. To that end we’ve added a channel called #fresh-on-master to announce new features, new abstractions, and API changes as they land. We’d love to get feedback on them before we release them. For example we are in the midst of landing some changes to our config system. I think it’s a big improvement in the API. I hope you all agree! Changes: In the config system,

Dict

has been renamed to `Shape`;

List

to `Array`;

Optional

to `Noneable`; and

PermissiveDict

to

Permissive

. The motivation here is to clearly delineate config use cases versus cases where you are using types as the inputs and outputs of solids as well as python typing types (for mypy and friends). We believe this will be clearer to users in addition to simplifying our own implementation and internal abstractions. Our recommended fix is not to used Shape and Array, but instead to use our new condensed config specification API. This allow one to use bare dictionaries instead of

Shape

, lists with one member instead of

Array

, bare types instead of

Field

with a single argument, and python primitive types (

int

,

bool

etc) instead of the dagster equivalents. These result in dramatically less verbose config specs in most cases. So instead of

from dagster import Shape, Field, Int, Array, String
	
     # ... code
     config=Shape({ # Dict prior to change
        'some_int' : Field(Int),
        'some_list: Field(Array[String]) # List prior to change
    })

one can instead write:

config={'some_int': int, 'some_list': [str]}

No imports and much simpler, cleaner syntax.

Hey all. Mind if I ask a newbie question?

I’m getting this error:

alembic.util.exc.CommandError: Path doesn't exist: '/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagster/core/storage/runs/sqlite/alembic/'.  Please use the 'init' command to create a new scripts folder.

I’m just following the Cereal tutorial

Max Gasner just published a new version: 0.6.7.post0.

Hey, I'm currently evaluating dagster as a replacement for airflow. I've listened to https://softwareengineeringdaily.com/2019/11/15/dagster-with-nick-schrock/ and could relate to many of the airflow issues you describe @schrockn. Thanks for your insights, very interesting episode 🙂 So, although we started developing our own k8s native tool, I'd like to give dagster a go since it sounded interesting when you described it and looked even better when I went through the tutorial. Is there any reason why you're using your own k8s launcher in dagster-k8s instead of Dask Kubernetes?

Hello guys! I started using Dagit in my project recently so have some questions about. I hope you will help me. So, I have the same error as Cam:

alembic.util.exc.CommandError: Path doesn't exist: '/home/newander/PycharmProjects/etl-backend/venv/lib/python3.8/site-packages/dagster/core/storage/runs/sqlite/alembic/'.  Please use the 'init' command to create a new scripts folder.

As I may see, this error was occurred by the next causes: I want to use postgresql as storage and add it to `repository.yaml`:

storage:
  run_storage:
    module: dagster_postgres.run_storage
    class: PostgresRunStorage
    config:
      postgres_url: "<postgresql://postgres:123>@localhost:5432/etl_backend"

  event_log_storage:
    module: dagster_postgres.event_log
    class: PostgresEventLogStorage
    config:
      postgres_url: "<postgresql://postgres:123>@localhost:5432/etl_backend"

But when I tried to start with

dagster schedule up -y repository.yaml

I received the error above. I think this error is about the

dagster/cli/schedule.py:157

code line where don't specified

config.yaml

and using default where specified SQL database. So, can you help me with it? (I can write pull request to fix CLI schedule up command but it will be too long) Aaand the next question: what settings I need to add into yaml to specify script/version directory for Alembic?

set the channel topic: General discussion and annoucements!

Hey all, just a quick question/suggestion on the latest addition to the static type check analysis relating to the source to destination input mappings. I’ve updated my code to use the latest version and am now getting an DagsterInvalidDefinitionError. This error is thrown when I try to pass the input of a composite solid into a nested solid within the composite where the composite solid’s input definition is of a specific DataFrame type but the nested solid is a generic DataFrame type. Since the sub-solid accepts a generic DataFrame type, I believe it should be legal to pass inputs from composite solid to it’s sub solids where a composite solid’s type maybe different from the sub solid and that the check happening just in time. This issue may be happening since I’m using a DataFrame which may need some inference logic to enforce the correct type validation. I could also be thinking about this wrong or not using the correct pattern, any suggestions on how to handle this scenario?

The online docs show how you can enable re-execution using prior runs' intermediates, from dagit. How would I enable re-execution using prior runs' intermediates from the python API (using execute_pipeline())?

I'm currently evaluating Dagster as a replacement for Airflow in our team. Currently we have multiple Git repositories that contain code for different aspects of our ETL/ML needs. Each Git-repo can contain multiple pipelines. These pipelines are run regularly in production, either on a schedule or some external trigger. I've been reading the docs, and know that Dagster has the concept of "repositories" to group multiple pipelines. In our case we'd probably define one Dagster-repo for each Git-repo? However, it's my understanding that when I launch dagit I can only point it to a single repo definition file at the time? Is there a way of pointing dagit at multiple repo-definition files? Or will we need to have a separate dagit instance for each Dagster/Git-repo? How do you recommend we'd structure our Dagster repos/pipelines?

reading through the docs, it seems like solid dependencies are very much centered around input/output dependencies. However, we have a few scenarios where the completion of a solid matters much more than the output. For example, a solid that needs to wait until another solids computation has completed, but doesn't need the output of said computation. Is there a pre-defined pattern for declaring dependencies between solids without needing to explicitly define inputs and outputs?