Hi Community!
I'm playing a bit with Dagster orchestrator and I have some doubts. My intention is use this tool only as a scheduler and runner for commands since I have the business logic developed in other repositories and containers. So, the architecture would be:
1. A container with Dagit
2. A container with Dagster daemon
3. A container with DBT models (business logic)
4. A container with Python runnable functions
Here my doubts:
• What would the best way be to define
ops and
jobs to execute DBT models (2) from the Dagster daemon environment (1)? Is there some specific Dagster library to make this easier? The command to run by the Dagster daemon should be translated to something similar to:
docker exec -t [container_DBT] bash -c "dbt run ...."
• The question to execute the Python scripts hosted in the other Docker container.
Thanks in advance!