Rubén Lopez Lozoya
03/11/2021, 12:05 PMDavid
03/11/2021, 1:09 PMLaura Moraes
03/11/2021, 1:17 PMLaura Moraes
03/11/2021, 1:19 PMSteve Pletcher
03/11/2021, 2:14 PMSteve Pletcher
03/11/2021, 2:34 PMMatej Války
03/11/2021, 3:17 PMMelle Minderhoud
03/11/2021, 4:29 PMUsage: dagster api execute_step [OPTIONS] INPUT_JSON
Try 'dagster api execute_step --help' for help.
Error: Got unexpected extra arguments ("ExecuteStepArgs", "instance_ref": {"__class__": "InstanceRef", "compute_logs_data": {"__class__": "ConfigurableClassData", "class_name": "LocalComputeLogManager", "config_yaml": "base_dir: /tmp/storage\n", "module_name": "dagster.core.storage.local_compute_log_manager"}, "custom_instance_class_data": null, "event_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteEventLogStorage", "config_yaml": "base_dir: /tmp/history/runs/\n", "module_name": "dagster.core.storage.event_log"}, "local_artifact_storage_data": {"__class__": "ConfigurableClassData", "class_name": "LocalArtifactStorage", "config_yaml": "base_dir: /tmp\n", "module_name": "dagster.core.storage.root"}, "run_coordinator_data": {"__class__": "ConfigurableClassData", "class_name": "DefaultRunCoordinator", "config_yaml": "{}\n", "module_name": "dagster.core.run_coordinator"}, "run_launcher_data": {"__class__": "ConfigurableClassData", "class_name": "DefaultRunLauncher", "config_yaml": "{}\n", "module_name": "dagster"}, "run_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteRunStorage", "config_yaml": "base_dir: /tmp/history/\n", "module_name": "dagster.core.storage.runs"}, "schedule_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteScheduleStorage", "config_yaml": "base_dir: /tmp/schedules\n", "module_name": "dagster.core.storage.schedules"}, "scheduler_data": {"__class__": "ConfigurableClassData", "class_name": "DagsterDaemonScheduler", "config_yaml": "{}\n", "module_name": "dagster.core.scheduler"}, "settings": {"backfill": null, "sensor_settings": null, "telemetry": null}}, "pipeline_origin": {"__class__": "PipelinePythonOrigin", "pipeline_name": "hello_cereal_pipeline", "repository_origin": {"__class__": "RepositoryPythonOrigin", "code_pointer": {"__class__": "ModuleCodePointer", "fn_name": "hello_cereal_pipeline", "module": "airflow_test.airflow"}, "container_image": null, "executable_path": "Library/Caches/pypoetry/virtualenvs/churn-metrics-ENa28q3B-py3.8/bin/python"}}, "pipeline_run_id": "manual__2021-03-11T15:04:26.729658+00:00", "retries_dict": {}, "should_verify_step": false, "step_keys_to_execute": ["hello_cereal"]})
Any advice on how we can solve this issue?
The DAG we are using is:Melle Minderhoud
03/11/2021, 4:29 PM'''
The airflow DAG scaffold for airflow_tests.airflow.hello_cereal_pipeline
Note that this docstring must contain the strings "airflow" and "DAG" for
Airflow to properly detect it as a DAG
See: <http://bit.ly/307VMum>
'''
import datetime
import yaml
from dagster_airflow.factory import make_airflow_dag_containerized
################################################################################
# #
# # This environment is auto-generated from your configs and/or presets
# #
################################################################################
ENVIRONMENT = '''
intermediate_storage:
filesystem:
config:
base_dir: /tmp/dagster-airflow/hello_cereal_pipeline
'''
################################################################################
# #
# # NOTE: these arguments should be edited for your environment
# #
################################################################################
DEFAULT_ARGS = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.datetime(2021, 3, 10),
'email': ['<mailto:airflow@example.com|airflow@example.com>'],
'email_on_failure': False,
'email_on_retry': False,
'auto_remove': False
}
dag, tasks = make_airflow_dag_containerized(
# NOTE: you must ensure that airflow_tests.airflow is
# installed or available on sys.path, otherwise, this import will fail.
module_name='airflow_test.airflow',
pipeline_name='hello_cereal_pipeline',
image = 'dagster-docker-test-cereal',
run_config=yaml.safe_load(ENVIRONMENT),
dag_kwargs={'default_args': DEFAULT_ARGS, 'max_active_runs': 1}
)
And the pipeline is defined as:
from dagster import pipeline, solid
@solid
def hello_cereal(context):
print('Hallo')
print("finished")
<http://context.log.info|context.log.info>('finished')
@pipeline
def hello_cereal_pipeline():
hello_cereal()
ps. The docs are refering to 'dagster_graphql', but I assume it should be dagster-graphql?Jeff Hulbert
03/11/2021, 4:34 PM@lambda_solid
def collect(lst: List) -> List:
return lst
@pipeline
def pipeline_fanin():
tables = ["table1", "table2", "table3"]
all_tables_run = []
for table in tables:
run_table = configured(
load_table, name=f"load_table_{table}"
)({"table_name": table})
all_tables_run.append(run_table())
collect(all_tables_run)
Andrew Brown
03/11/2021, 5:36 PM.env
file, which is loaded by docker-compose.yml
. I have also verified the $DAGSTER_CURRENT_IMAGE env var is set in the pipelines container by shelling into it while it's running.
Here are the running containers output by `docker ps`:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9360afd792f3 dagster-data-orchestration_dagster_daemon "dagster-daemon run" 18 hours ago Up 18 hours dagster_daemon
606fbbba144c dagster-data-orchestration_dagster_dagit "dagit -h 0.0.0.0 -p…" 18 hours ago Up 18 hours 0.0.0.0:3000->3000/tcp dagster_dagit
85b25377d7c3 postgres:11 "docker-entrypoint.s…" 18 hours ago Up 18 hours 0.0.0.0:5432->5432/tcp dagster_postgresql
51c5d0f47b5b dagster-data-orchestration_dagster_pipelines "dagster api grpc -h…" 18 hours ago Up 18 hours 0.0.0.0:4000->4000/tcp dagster_pipelines
And here is the value of $DAGSTER_CURRENT_IMAGE in dagster-data-orchestration_dagster_pipelines
's running container (retrieved via bash in running container docker exec -it 51c5d0f47b5b /bin/sh
) :
# echo $DAGSTER_CURRENT_IMAGE
dagster-data-orchestration_dagster_pipelines
Deveshi
03/11/2021, 5:49 PMKing Chung Huang
03/11/2021, 6:30 PMdagster-dask
module development to implement changes/improvements needed in my group. I'm going to hit dagster-io/dagster#2901 and dagster-io/dagster#3829, first, then clean up the languishing PRs from the fall.Adrian
03/11/2021, 8:45 PMDaniel Kim
03/11/2021, 9:30 PMYan
03/11/2021, 11:22 PMNoah K
03/12/2021, 12:37 AMrecompress.configured({'draco_compress': False}, name='recompress_without_draco')(asset)
) is there a reason the name is required and it doesn't just do the automatic _2
like when normally multi-instantiating solids?Noah K
03/12/2021, 12:38 AMNoah K
03/12/2021, 12:38 AMNoah K
03/12/2021, 12:42 AMJosh Taylor
03/12/2021, 1:49 AMdagster.core.errors.DagsterStepOutputNotFoundError: Core compute for solid "files_in_directory" did not return an output for non-optional output "result"
King Chung Huang
03/12/2021, 2:08 AMdocs/sphinx/sections/api/apidocs/libraries/dagster_dask.rst
. And, following the directions at https://docs.dagster.io/community/contributing#developing-docs, I can run the docs website locally. But, when I navigate to the page, I see the original page without my local changes.
I tried make buildnext
, but it fails because there's no target named buildnext
in the Makefile.David
03/12/2021, 9:04 AMMarco
03/12/2021, 9:22 AMsashank
03/12/2021, 4:57 PMsashank
03/12/2021, 4:59 PMTobias Macey
03/12/2021, 7:02 PMbklau-zap
03/12/2021, 7:15 PMTobias Macey
03/12/2021, 8:53 PMRubén Lopez Lozoya
03/13/2021, 4:08 PM