Airflow ingest We've expanded the tooling available to teams with existing Airflow installations that are interested in incrementally adopting Dagster. Previously, we provided only injection tools that allowed developers to write Dagster pipelines and then compile them into Airflow DAGs for execution. We've now added ingestion tools that allow teams to move to Dagster for execution without having to rewrite all of their legacy pipelines in Dagster. In this approach, Airflow DAGs are kept in their own container/environment, compiled into Dagster pipelines, and run via the Dagster orchestrator. See the

airflow_ingest

example for details!

I'm trying out the migration to v0.8 now and wanted to seek some clarifications. when running dagster in a k8s cluster with celery workers, if I call

execute_pipeline

or

execute_pipeline_iterator

, which worker is the child pipeline executed in? In the same worker? Or will it be enqueued as a new job in the broker queue, to be picked up later by another worker? I'm assuming it's the former.

Which parts of the docs have you found the most useful for learning about Dagster?

Hi! I wanted to ask you how do you handle dagster logs that get too big? How would for example a clean of old logs/events/etc look like? In only under a month we are seeing that the dagster db take huge space and becoming slower probably due to this

Sandy Ryza just published a new version: 0.8.1.

Hi guys, I just recently (few days ago) started a project based on Dagster. I read a lot of docs and started to write some simple tests. Now I see 8.0 version is out. Do you think it's better for me to immediately switch to 8 as we are at the start and there is no code to update anyway. I am asking because I am not sure when docs for 8.0 will be updated as I now completely rely on the docs for all my progress? Thanks.

Hi, is anyone else having trouble with the schedules? when trying to show the schedules these don't appear in dagit

My repo.py is structured like this,

Hey, thanks for all your help in the migration process! I have a new issue with the schedules and would like to ask for your help again. My project structured as follows:

pipelines/
--app/
----twitter/
------some_module.py
----workspaces.yaml
----repository.py
--home/
Pipifile
run.sh
...

To run dagit I use a bash script that adds

pipelines/app

to the

PYTHONPATH

so that the modules inside

app

can be found. It used to work also with the scheduled executions. With the migration to 0.8 the schedules are failing silently. However looking into the

results

file that the scheduler generates i found this error (please, apologize me for the formatting)

{"__class__": "IPCErrorMessage", "message": null, "serializable_error_info": {"__class__": "SerializableErrorInfo", "cause": null, "cls_name": "ModuleNotFoundError", "message": "ModuleNotFoundError: No module named 'twitter'\n", "stack": ["  File \"/home/user/.local/share/virtualenvs/user-pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/serdes/ipc.py\", line 116, in ipc_write_stream\n    yield FileBasedWriteStream(file_path)\n", "  File \"/home/user/.local/share/virtualenvs/user-pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/cli/api.py\", line 492, in launch_scheduled_execution\n    repo_def = recon_repo.get_definition()\n", "  File \"/home/user/.local/share/virtualenvs/user-pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/core/definitions/reconstructable.py\", line 36, in get_definition\n    return repository_def_from_pointer(self.pointer)\n", "  File \"/home/user/.local/share/virtualenvs/user-pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/core/definitions/reconstructable.py\", line 311, in repository_def_from_pointer\n    target = def_from_pointer(pointer)\n", "  File \"/home/user/.local/share/virtualenvs/user-pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/core/definitions/reconstructable.py\", line 271, in def_from_pointer\n    target = pointer.load_target()\n", "  File \"/home/user/.local/share/virtualenvs/user-pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/core/code_pointer.py\", line 84, in load_target\n    module = load_python_file(self.python_file)\n", "  File \"/home/user/.local/share/virtualenvs/user-pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/core/code_pointer.py\", line 73, in load_python_file\n    return import_module_from_path(module_name, python_file)\n", "  File \"/home/user/.local/share/virtualenvs/user-pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/seven/__init__.py\", line 82, in import_module_from_path\n    spec.loader.exec_module(module)\n", "  File \"<frozen importlib._bootstrap_external>\", line 783, in exec_module\n", "  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n", "  File \"/home/user/user-pipelines/app/repository.py\", line 2, in <module>\n    from twitter.pipelines import (\n"]}}
{"__class__": "IPCEndMessage"}

It seems that the modules are not being picked well. Do you have advise on how, and where to setup the pythonpath so that the imports work? I guess this has to do with the new execution architecture?

prha just published a new version: 0.8.2.

Hi all - how can we join the webinar ?

Hi guys I'm having an issue with schedules. I have a repo setup like this:

@repository
def repo():
    return [
        test,
        ScheduleDefinition(
            name="test_schedule",
            cron_schedule="* * * * *",
            pipeline_name="test",
        )
    ]

And my

dagster.yaml

looks like that:

scheduler:
  module: dagster_cron.cron_scheduler
  class: SystemCronScheduler

Looking at crontab and dagit, it looks like everything is setup alright, yet my test pipeline isn't running at all... Any help would be much appreciated!

We will be starting webinar about 5 mins late

I have to reset the breaker in my home office

Perils of remote work

For anyone that wants to join: https://us02web.zoom.us/j/82684684275?pwd=cDE0NzdWN2w4Z1QwdDFlOEtnZ1gwUT09

Thanks so much all! That was really fun and a great turnout with really incisive questions.

Feel free to keep the convo going in here!

youtube video for posterity:

twitter thread about it

👋 I’ve been diving into using dagster-aws and pyspark for use on EMR. I currently orchestrate and deploy to an airflow instance with dags in a zip file which seems to be somewhat problematic when using the emr pyspark step resource and having the dag be in a zip. Wondering if that’s on the radar or maybe I should ticket out more detail?

Hello! 👋 I would like to ask about docker-compose file. Is there any available for dagster? I would like to test solution quickly on my local cluster using KIND

Hi everyone! I've updated my dagster setup to 0.8.2 and found that there is no reload button anymore and at the same time dagit instance doesn't watch files so I have to redeploy my containers or run dagit and celery workers upder pm2 or any other watching manager alternative. Is this correct or am I missing something? Why have you eliminated the reload functionality?

Another strange behavior with celery: it seems not to parallelize execution though I have 2 workers

One question, is there any branch/issue that I can track to follow the development of the containerised environments? Very excited about this one!

Nate Kupp just published a new version: 0.8.3.