Posted on

C++17: Efficiently Returning std::vector from Functions

The discussion centers on returning std::vector from C++ functions, highlighting Return Value Optimization (RVO) introduced in C++17. RVO allows the compiler to avoid copying vectors by constructing them in place when there's a single return path. For multiple return paths, std::move is used to transfer ownership efficiently. Exceptions exist, particularly with the conditional operator, which requires copying. Returning references from member functions is safer than from free functions since the object's lifetime ensures validity.

Posted on

Mastering DataOps: Orchestrating AWS Glue Workflows

The implemented stages of ingestion, preprocessing, EDA, and feature engineering have transitioned to automation and monitoring, forming a cohesive DataOps layer. By introducing orchestration, the independent Glue jobs become an automated, reliable workflow. Testing confirmed successful execution, paving the way for regular automations to enhance operations and insights from data.

Posted on

Training and Evaluating ML Models with AWS Glue

This post details the development of a Machine Learning Pipeline for demand forecasting. Utilizing AWS Glue and PySpark, it covers training and evaluating Linear Regression and Random Forest models using an engineered feature dataset. Results show Random Forest slightly outperforms Linear Regression, demonstrating effective model stability and reliability for deployment.

Posted on

Mastering Feature Engineering for Machine Learning

The Feature Engineering stage follows Exploratory Data Analysis, preparing the dataset for machine learning. It generates temporal and statistical features, encodes categorical identifiers, and ensures schema consistency. Implemented in AWS Glue, it enables reproducibility and scalability for model training, enhancing forecasting accuracy by incorporating lag and rolling average features.

Posted on

Mastering EDA for Demand Forecasting on AWS

This article expands on a previous post about building a serverless ETL pipeline on AWS by focusing on Exploratory Data Analysis (EDA). It details how to establish the EDA environment using AWS Glue and PySpark after cleaning the dataset. Key insights include sales trends, store and item performance, and correlation analysis, laying the groundwork for a demand forecasting model.

Posted on

Enhancing Your ETL Pipeline with AWS Glue and PySpark

The post details enhancements made to a serverless ETL pipeline using AWS Glue and PySpark for retail sales data. Improvements include explicit column type conversions, missing value imputation, normalization of sales data, and integration of logging for observability. These changes aim to create a production-ready, machine-learning-friendly preprocessing layer for effective data analysis.

Posted on

Building an ETL Pipeline for Retail Demand Data

This project aims to develop a demand forecasting solution for retail using historical sales data from Kaggle. A data pipeline employing AWS Glue and PySpark will preprocess the data by cleaning and splitting it into training and testing sets. The objective is to maximize inventory management and customer satisfaction.

Posted on

How Did I Run and Containerise My First Flask App?

The article discusses the challenges of consistent application behavior in software development and how Docker addresses these issues. It outlines the creation of a simple Flask app, its containerization using Docker, and steps to ensure accessibility from outside the container. Troubleshooting and cleanup procedures are also covered, emphasizing a portable setup.

Posted on

Introduction to Multi-Threaded Programming: Key Concepts

This blog post discusses how multi-tasking enables efficient CPU time-sharing among programs, allowing them to seemingly run simultaneously on a single-core processor. The OS scheduler manages task switching, allowing programs like a music player and a word processor to share CPU time effectively. Context switching is a rapid process that gives the appearance of parallel execution. However, distinct processes have isolated memory spaces, complicating data sharing. Threads within a process, on the other hand, share address space, simplifying communication and resource management. This post also introduces the pthread library for creating threads in C, showcasing the practicality of multi-threading.

Posted on

Understanding Parallelism in Uni-Processor Systems

The content explains that a uni-processor system has only one CPU, which can execute only one piece of code at a time. This leads to pseudo parallelism, where multiple programs seem to run simultaneously by sharing CPU time. For illustration, two simple programs are presented: one continuously prints "Hello World" and the other prints "Hello Boss." In practice, they take turns using the CPU, facilitated by the operating system's scheduler. The blog emphasizes terminologies like process and infinite loop, providing insights into how parallelism works, even in environments with limited processing capabilities.

Posted on

How to Fix AWS SignatureDoesNotMatch Error

The "SignatureDoesNotMatch" error often occurs when uploading files to AWS S3 due to signature mismatches related to secret keys. The author shares a step-by-step guide to troubleshoot this issue, which includes verifying IAM user credentials, configuring access keys, and successfully retrying the upload operation after resolving permissions.

Posted on

Parallel Processing: Best Data Partitioning Strategies for Maximum Efficiency

Efficient parallel processing relies on smart data partitioning strategies to distribute workloads across multiple processors. This blog explores three fundamental techniques: Block Partitioning, Cyclic Partitioning, and Block-Cyclic Partitioning. Through step-by-step explanations and visual diagrams, you'll learn how these methods optimize performance by balancing data distribution. Whether you're new to parallel computing or looking to refine your understanding, this guide breaks down complex concepts into simple, digestible insights. Stay tuned for a practical implementation using MPI (Message Passing Interface) in the next section!