I deploy Dagster in kubernetes with a separate User Code Repository and Infrastructure Deployment (Daemon and Webserver). When I execute a run Dagster starts a new kubernetes Job and logs the output in the job. How can I access the logs using Loki? I see the logs of my User Code Repository as this is a normal Deployment but I can't find the logs of the job.

We currently don't have a direct integration with Loki for log management. However, you can set up your Kubernetes cluster to forward logs to Loki, including those from Dagster jobs. Here's a general approach you can take, based on Kubernetes logging practices: 1. Configure your Kubernetes cluster to use Loki: You'll need to set up Loki as your log aggregation system. This typically involves deploying Loki to your cluster and configuring your logging driver to send logs to Loki. The Grafana website provides documentation on how to do this. 2. Ensure your Dagster job pods output logs to stdout/stderr: By default, Kubernetes captures logs written to stdout and stderr by the containers in your pods. Make sure that your Dagster jobs are configured to log to these streams so that Kubernetes can capture the logs. 3. Use Promtail or a similar agent to ship logs: Deploy Promtail, Fluentd, or a similar agent to your cluster. These agents will collect the logs from your pods and ship them to Loki. You'll need to configure the agent with the correct scraping rules to find and forward the logs from your Dagster job pods. 4. Query logs in Loki: Once your logs are being shipped to Loki, you can query them using Loki's query language in the Grafana dashboard. Remember that the specifics of how you set up logging with Loki will depend on your particular Kubernetes environment and configuration. You may need to adjust your setup based on factors like your cluster's network policies, RBAC settings, and how you've deployed Loki and your log shipping agent. For more detailed instructions and best practices on setting up logging in Kubernetes with Loki, you should refer to the official documentation provided by Grafana and the Kubernetes community.