Posted on

C++17: Efficiently Returning std::vector from Functions

The discussion centers on returning std::vector from C++ functions, highlighting Return Value Optimization (RVO) introduced in C++17. RVO allows the compiler to avoid copying vectors by constructing them in place when there's a single return path. For multiple return paths, std::move is used to transfer ownership efficiently. Exceptions exist, particularly with the conditional operator, which requires copying. Returning references from member functions is safer than from free functions since the object's lifetime ensures validity.

Posted on

Mastering DataOps: Orchestrating AWS Glue Workflows

The implemented stages of ingestion, preprocessing, EDA, and feature engineering have transitioned to automation and monitoring, forming a cohesive DataOps layer. By introducing orchestration, the independent Glue jobs become an automated, reliable workflow. Testing confirmed successful execution, paving the way for regular automations to enhance operations and insights from data.

Posted on

Training and Evaluating ML Models with AWS Glue

This post details the development of a Machine Learning Pipeline for demand forecasting. Utilizing AWS Glue and PySpark, it covers training and evaluating Linear Regression and Random Forest models using an engineered feature dataset. Results show Random Forest slightly outperforms Linear Regression, demonstrating effective model stability and reliability for deployment.

Posted on

Mastering Feature Engineering for Machine Learning

The Feature Engineering stage follows Exploratory Data Analysis, preparing the dataset for machine learning. It generates temporal and statistical features, encodes categorical identifiers, and ensures schema consistency. Implemented in AWS Glue, it enables reproducibility and scalability for model training, enhancing forecasting accuracy by incorporating lag and rolling average features.

Posted on

Enhancing Your ETL Pipeline with AWS Glue and PySpark

The post details enhancements made to a serverless ETL pipeline using AWS Glue and PySpark for retail sales data. Improvements include explicit column type conversions, missing value imputation, normalization of sales data, and integration of logging for observability. These changes aim to create a production-ready, machine-learning-friendly preprocessing layer for effective data analysis.

Posted on

Building an ETL Pipeline for Retail Demand Data

This project aims to develop a demand forecasting solution for retail using historical sales data from Kaggle. A data pipeline employing AWS Glue and PySpark will preprocess the data by cleaning and splitting it into training and testing sets. The objective is to maximize inventory management and customer satisfaction.

Posted on

Building a Real-Time Aircraft Tracking System with AWS Lambda, Kinesis, and DynamoDB

Aviation data has always been fascinating. Planes crisscross the globe. Each one sends out tiny bursts of information as it soars through the sky. Thanks…

Continue reading → Building a Real-Time Aircraft Tracking System with AWS Lambda, Kinesis, and DynamoDB

Posted on

Parallel Processing: Best Data Partitioning Strategies for Maximum Efficiency

Efficient parallel processing relies on smart data partitioning strategies to distribute workloads across multiple processors. This blog explores three fundamental techniques: Block Partitioning, Cyclic Partitioning, and Block-Cyclic Partitioning. Through step-by-step explanations and visual diagrams, you'll learn how these methods optimize performance by balancing data distribution. Whether you're new to parallel computing or looking to refine your understanding, this guide breaks down complex concepts into simple, digestible insights. Stay tuned for a practical implementation using MPI (Message Passing Interface) in the next section!