won
11/09/2021, 1:42 PMdocker_postgresql
, docker_user_code
, docker_dagit
, docker_daemon
)
See in UI my jobs.
Next go to lachpad and run any of jobs.
dagster starting new container with same image as my user_code container.
Then i get error:
dagster.core.errors.DagsterInvariantViolationError: Could not find pipeline 'ozon_fbs_job'. Found: 'my_job'.
File "/usr/local/lib/python3.7/site-packages/dagster/grpc/impl.py", line 75, in core_execute_run
recon_pipeline.get_definition()
File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/reconstructable.py", line 112, in get_definition
defn = self.repository.get_definition().get_pipeline(self.pipeline_name)
File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/repository.py", line 878, in get_pipeline
return self._repository_data.get_pipeline(name)
File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/repository.py", line 658, in get_pipeline
return self._pipelines.get_definition(pipeline_name)
File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/repository.py", line 141, in get_definition
for found_name in self.get_definition_names()
I haven't got job with my_job
name 🤯
Where is MY jobs? (ozon_stocks_job
, ozon_fbs_job
) 👀
project struct:
dagster_docker/
├───user_code/
│ ├───ops/
│ │ └───ozon_fbs.py
│ │ └───ozon_stocks.py
│ ├───resoutces/
│ │ └───db_resource.py
│ └───repo.py
├───dagster.yaml
├───docker-compose.yml
├───Dockerfile_dagster
├───Dockerfile_user_code
├───requirements.txt
├───workspace.yaml
Wher did i miss?from dagster import repository, schedule
from ops.ozon_fbs import ozon_fbs_job
from ops.ozon_stocks import ozon_stocks_job
@schedule(cron_schedule="1 10 * * *", job=ozon_stocks_job, execution_timezone="US/Central")
def my_schedule(_context):
return {}
@schedule(cron_schedule="0 10 * * *", job=ozon_fbs_job, execution_timezone="US/Central")
def ozon_daily_schedule(_context):
run_config = {}
return run_config
@repository
def deploy_docker_repository():
return [ozon_stocks_job, ozon_fbs_job, my_schedule, ozon_daily_schedule]
scheduler:
module: dagster.core.scheduler
class: DagsterDaemonScheduler
run_coordinator:
module: dagster.core.run_coordinator
class: QueuedRunCoordinator
run_launcher:
module: dagster_docker
class: DockerRunLauncher
config:
env_vars:
- DAGSTER_POSTGRES_USER
- DAGSTER_POSTGRES_PASSWORD
- DAGSTER_POSTGRES_DB
container_kwargs:
volumes:
- repo.py:/opt/dagster/app/
network: docker_network
run_storage:
module: dagster_postgres.run_storage
class: PostgresRunStorage
config:
postgres_db:
hostname: docker_postgresql
username:
env: DAGSTER_POSTGRES_USER
password:
env: DAGSTER_POSTGRES_PASSWORD
db_name:
env: DAGSTER_POSTGRES_DB
port: 5432
schedule_storage:
module: dagster_postgres.schedule_storage
class: PostgresScheduleStorage
config:
postgres_db:
hostname: docker_postgresql
username:
env: DAGSTER_POSTGRES_USER
password:
env: DAGSTER_POSTGRES_PASSWORD
db_name:
env: DAGSTER_POSTGRES_DB
port: 5432
event_log_storage:
module: dagster_postgres.event_log
class: PostgresEventLogStorage
config:
postgres_db:
hostname: docker_postgresql
username:
env: DAGSTER_POSTGRES_USER
password:
env: DAGSTER_POSTGRES_PASSWORD
db_name:
env: DAGSTER_POSTGRES_DB
port: 5432
docker_user_code:
build:
context: .
dockerfile: ./Dockerfile_user_code
container_name: docker_user_code
image: docker_user_code_image
restart: always
environment:
DAGSTER_POSTGRES_USER: "postgres_user"
DAGSTER_POSTGRES_PASSWORD: "postgres_password"
DAGSTER_POSTGRES_DB: "postgres_db"
DAGSTER_CURRENT_IMAGE: "docker_user_code_image"
volumes: # Make docker client accessible so we can terminate containers from dagit
- ./user_code:/opt/dagster/app/
networks:
- docker_network
Dockerfile_user_code
FROM python:3.7-slim
RUN pip install \
dagster \
dagster-postgres \
dagster-docker
COPY requirements.txt /requirements.txt
RUN pip install --no-cache-dir --requirement /requirements.txt && rm requirements.txt
# Add repository code
WORKDIR /opt/dagster/app
COPY user_code/repo.py /opt/dagster/app
# Run dagster gRPC server on port 4000
EXPOSE 4000
# CMD allows this to be overridden from run launchers or executors that want
# to run other commands against your repository
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "repo.py"]
max
11/09/2021, 5:08 PMdaniel
11/09/2021, 5:11 PMwon
11/09/2021, 7:53 PMdocker-compose up -d --no-deps --build docker_user_code
fix my trouble.
I have another question.
After job is done - container exited but still exist.
What should i do to make them automatically remove?max
11/09/2021, 7:56 PM--rm
when you run docker-compose
daniel
11/09/2021, 10:41 PMrun_launcher:
module: dagster_docker
class: DockerRunLauncher
config:
container_kwargs:
auto_remove: true
won
11/10/2021, 6:12 AM