The implemented stages of ingestion, preprocessing, EDA, and feature engineering have transitioned to automation and monitoring, forming a cohesive DataOps layer. By introducing orchestration, the independent Glue jobs become an automated, reliable workflow. Testing confirmed successful execution, paving the way for regular automations to enhance operations and insights from data.
This post details the development of a Machine Learning Pipeline for demand forecasting. Utilizing AWS Glue and PySpark, it covers training and evaluating Linear Regression and Random Forest models using an engineered feature dataset. Results show Random Forest slightly outperforms Linear Regression, demonstrating effective model stability and reliability for deployment.
The Feature Engineering stage follows Exploratory Data Analysis, preparing the dataset for machine learning. It generates temporal and statistical features, encodes categorical identifiers, and ensures schema consistency. Implemented in AWS Glue, it enables reproducibility and scalability for model training, enhancing forecasting accuracy by incorporating lag and rolling average features.
The post details enhancements made to a serverless ETL pipeline using AWS Glue and PySpark for retail sales data. Improvements include explicit column type conversions, missing value imputation, normalization of sales data, and integration of logging for observability. These changes aim to create a production-ready, machine-learning-friendly preprocessing layer for effective data analysis.
This project aims to develop a demand forecasting solution for retail using historical sales data from Kaggle. A data pipeline employing AWS Glue and PySpark will preprocess the data by cleaning and splitting it into training and testing sets. The objective is to maximize inventory management and customer satisfaction.
Aviation data has always been fascinating. Planes crisscross the globe. Each one sends out tiny bursts of information as it soars through the sky. Thanks…
Efficient parallel processing relies on smart data partitioning strategies to distribute workloads across multiple processors. This blog explores three fundamental techniques: Block Partitioning, Cyclic Partitioning, and Block-Cyclic Partitioning. Through step-by-step explanations and visual diagrams, you'll learn how these methods optimize performance by balancing data distribution. Whether you're new to parallel computing or looking to refine your understanding, this guide breaks down complex concepts into simple, digestible insights. Stay tuned for a practical implementation using MPI (Message Passing Interface) in the next section!
Tech For Talk 