https://dagster.io/ logo
#ask-ai
Title
# ask-ai
p

Peter Mburu

02/21/2024, 10:24 AM
How to log tqdm progress in dagster
s

Scout

02/21/2024, 10:24 AM
In Dagster, logging is an essential part of the execution context, and you can use the built-in log manager,
context.log
, to log messages at various levels such as
debug
,
info
,
warning
,
error
, and
critical
. However, integrating a progress bar like
tqdm
directly into the Dagster logs requires a bit of customization. Based on the documentation and GitHub issues, there has been interest in integrating progress bars like
tqdm
with Dagster, especially for long-running tasks. Users have expressed a desire to see progress from the UI, and while there isn't a built-in feature for progress bars in Dagster as of the last update, there are some workarounds that have been discussed in the community. One approach mentioned in the GitHub issues is to periodically log to the structured event log, which will show up in real-time in the UI. This can be done by embedding
context.log.debug()
or another appropriate logging method within your generator or loop that is processing the data. This method does not create a traditional progress bar but can provide updates on the progress of the task. Here's an example of how you might log progress using
tqdm
and `context.log`:
Copy code
python from tqdm import tqdm from dagster import op  @op def my_op(context):     for _ in tqdm(range(100)):         # Your processing logic here         <http://context.log.info|context.log.info>("Processed an item")
In this example,
tqdm
is used to wrap the iterable, and
<http://context.log.info|context.log.info>
is used to log the progress. However, this might not result in a traditional progress bar in the logs but will provide periodic updates on the progress. Additionally, there have been discussions about issues with
tqdm
causing the Dagit UI to become unresponsive when a large number of logs are generated. A workaround suggested in the GitHub issues is to use alternatives like
tqdm_telegram
or
tqdm_slack
to send progress updates to external services instead of logging them directly in Dagit. It's important to note that while these methods can provide progress updates, they may not offer the same experience as a dedicated progress bar in the UI. If you require a more integrated solution, you may need to explore custom implementations or wait for future updates that might add this feature to Dagster.
16 Views