Materializing Lightning Clustering pipeline ==================================== In my latest post, I perform Exploratory Data Analysis on NOAA's GLM dataset. I create a data pipeline in which I ingest the data and utilize #clustering analysis to create a rough map of lightning clusters . In the EDA #notebook, I load spatial, latitude and longitude raw data and implement k-means clustering algorithm on the test dataset. I evaluate the perform of the 'k' clusters and create a time lapse of lightning clusters. I convert the notebook into a Dagster orchestrated #data pipeline composed of materialized assets performing the required data ingestion and clustering analysis. https://blog.devgenius.io/exploratory-data-analysis-with-lightning-clustering-pipeline-6a2bca17d0d3 https://github.com/BayoAdejare/lightning-map https://github.com/BayoAdejare/lightning-map/blob/master/notebooks/eda_kmeans.ipynb #machinelearning #analytics #orchestration #data #pipeline #dagster #dataengineering #etl

Hey Adebayo - We have a dagster community call coming up on June 12th- would you be interested in showcasing this?