Hello, is it possible to use tags to filter solids? I'm using DBT and it would be wonderful to be able to filter solids with something like

tag:key=value*

Hey, I just noticed that when using lambda solids, I am unable to mock the function wrapped by the lambda solid (using unittest.mock). I've tried replacing the lambda_solid with a solid with @solid wrapper that just calls the function and the mock works fine there. Is there anything I am missing here? Can anyone explain this behavior?

Hi all! Our next Dagster monthly community meeting is tomorrow on Tuesday, March 9 at 9 am Pacific Time (UTC-7).  Link to Agenda Our 0.11.0 release is coming soon, so this month’s community meeting will feature presentations from Dagster’s core team about some of the features you will find in the new release. We have four presentations scheduled for the meeting: • @alex will present dynamic orchestration and the new

collect

step • @owen will discuss upcoming plans for the asset system, including asset lineage • @sandy  will will demo the new backfill and partition features in Dagit • @sashank will demo the new documentation infrastructure and design There will also be a Q&A and discussion section for any questions you have. If you’d like to submit a question in advance, feel free to do so here: Questions Form DM me your email or fill out this form to be added to the calendar invite: https://forms.gle/K2dni4iPq48GnUEq6. If you aren’t able to make it, we’ll share a recording afterward.

@sashank Is there a zoom link for above Dagster event?

Zoom link is on the calendar invite, but here’s the direct link: https://us02web.zoom.us/j/88939083535

When running a backfill we would potentially like to be able to change some solid behaviour accordingly. I haven't been able to see how we would be able to detect, in a clean way, at runtime that we're running a backfill. I can see that there is dagster_meta data emitted with pipeline and step events that we could potentially use, but it seems a bit private and maybe not to be relied upon in the future. Can anyone advise on the appropriate strategy or some obvious thing I may have missed?

Hi! I have such interesting dragster's sensor behavior. If I use function call inside sensor - it doesn't work at all, but if I use lambda - works. Is it an error or it's my error? Thank you

hello

thanks for the awesome work!

how do I define redis as result_backend and broker_url for the celery workers ?

nvm got it working created a celery_worker.yaml

was running it locally with dagster-celery

Hi All, probably a very silly question here. How do I use a placeholder for DB password (using redshift) in the yaml file? Plan is to avoid password in plain text and use a package like 1Password to retrieve it during run time? redshift: config: host: host_name port: 5439 user: user password: placeholder_password database: dbname

Hi everyone! Our community meeting is going to start in just a minute. Here's the direct Zoom link if you don't have it: https://us02web.zoom.us/j/88939083535

great community meeting as always - nice work <!subteam^S017AHZGB5Y|@elementl-team>! dagsir

My daemon stopped out of nowhere. Is there a place where I can see the daemon logs? Not the pipeline logs.

I'm trying to extend the docker example (https://github.com/dagster-io/dagster/tree/master/examples/deploy_docker) by making a simple solid that stores data as csv in the local file system. However, given that the containers that run the pipeline are ephimeral, how can I do it so that I can somehow retrieve the csv generated by my pipeline?

Responding out of thread since this is asked a lot, the new io managers system allows for this nicely 🙂

When launching a scheduled pipeline is the return from the repository fixed from the time you launch the scheduler? More specifically, if I want to do something like

@repository
def tmp_pipeline_repository():
    tmp_pipeline = pipeline_factory()
    return [tmp_pipeline, tmp_pipeline_schedule]

where

pipeline_factory

returns a pipeline whose functionality changes based on some external data which changes asynchronously from the pipeline, will the pipeline be reconstructed before each scheduled run? If not, is there a way to do this that conforms to

dagster

design patterns? Happy to clarify the details further if this is too vague

Hello all, I executed pipeline using

dagster.execute_pipeline

. The execution, however wasn't displayed in dagit pipeline's runs list. Is there anything I can do to fix this?

Hi all! A quick question regarding a potential issue I've come across with the dbt library - it seems like the

dbt_cli_snapshot

solid has incorrect configs. On line 340 here, you can see that

dbt snapshot

is set with options

thread

,

exclude

and

models

... however that command does not accept the

models

flag, but rather uses the

select

flag to choose which snapshot(s) to run (confirmed here). Does that look right to others as well? Anything I'm missing there or is this just one thing that needs to be fixed?

Hey everyone! 👋 I have a quick question about dynamic orchestration. I have the following pipeline, which simply gathers “files from inside of files” and indexes them into a database:

@pipeline
def ingestion_pipeline():
    def ingest(file):
        ingest_file(path=file)

    def spread(file):
        gather_files(path=file).map(ingest)

    gather_files().map(spread)

When I execute this pipeline, I get the following:

Solid "ingest_file" cannot be downstream of more than one dynamic output. It is downstream of both "gather_files" and "gather_files_2"

As I understand it, nesting dynamic outputs is not currently implemented. Alright, so instead of mapping from inside the

spread

function, I tried to map over its result, like this:

@pipeline
def ingestion_pipeline():
    def ingest(file):
        ingest_file(path=file)

    def spread(file):
        return gather_files(path=file)

    spread_files = gather_files().map(spread)
    spread_files.map(ingest)

It does not work either, which makes sense since the underlying dependency graph should be the same. Does anyone have tips on how to overcome this kind of situation using Dagster?

nesting dynamic outputs would be nice

Hey folk 👋 Question regarding `dagster-pandas`; it does not seem like there currently is support for the concept of Index in the pandas dataframe dagster type. Is there a workaround for this, and/or is this on the roadmap to add at some point? For context, I have an IO manager that outputs a dataframe to a RDMS table and uses the dataframe

index

to understand which column is the primary key (and uses it as the

merge

key). It looks like my only option right now would be to exclude that “column” from the pandas dagster type, though I would prefer to have the dagster validation on it as well (i.e.

non_nullable

,

unique

, plus the nice-to-have documentation!)

Hey everybody! We are currently looking to integrate Dagster, but we are currently facing some birth issues 🙂 I can't seem to get the idea of how the flow is stored and invoked from the docs. Is there any sync with the local machine repositories/workspaces with the remote machine host workspaces/repositories? How can I sync local pipeline which I created on my machine with the remote machine (main Dagster instance, used for production)? Sorry for the misunderstanding and thanks for the help! ✌️

goPuff is interested in getting a presentation from the Dagster team (we’re migrating away from Airflow). Who would be a good point of contact?

Hi all! I'm evaluating Dagster for two mid-sized projects I'm working on. In both cases we may use Dagster to orchestrate tasks and interactions between Sagemaker and S3, and dbt models on Snowflake. Has anyone had experience running SageMaker jobs (e.g. run a notebook, Sagemaker batch prediction transforms, etc.) via Dagster? If so, curious if there's any docs or lessons learned from the experience!

Is this the right place to ask a question or is there like a #help channel?

Hi probably another silly question here, I am looking to capture the Dagster console logs and dump them into a table, what is the best way to achieve this? I was wondering if there is an attribute I could use in place of console ? Many thanks!