Cloud / AWS / DevOps
Hands-on engineering guides covering AWS services, Glue ETL, Lambda automation, serverless architecture, distributed data pipelines, DevOps tooling, monitoring, infrastructure management, and cloud-native design patterns for scalable systems.
The blog discusses the integration of Amazon EventBridge to automate AWS Glue workflows every two minutes, enhancing operational efficiency in data engineering and machine learning tasks. It details steps to create and configure EventBridge rules, set permissions, and verify workflows, emphasizing improvements in responsiveness, agility, and DataOps maturity.
The implemented stages of ingestion, preprocessing, EDA, and feature engineering have transitioned to automation and monitoring, forming a cohesive DataOps layer. By introducing orchestration, the independent Glue jobs become an automated, reliable workflow. Testing confirmed successful execution, paving the way for regular automations to enhance operations and insights from data.
The post discusses the evolution of a data pipeline, highlighting the integration of an API-driven layer for enhanced observability. This new functionality allows authorized users to access real-time operational status without manual checks across AWS services. The approach improves transparency, accountability, and agility while enabling proactive monitoring and automated responses in future enhancements.
This post details the development of a Machine Learning Pipeline for demand forecasting. Utilizing AWS Glue and PySpark, it covers training and evaluating Linear Regression and Random Forest models using an engineered feature dataset. Results show Random Forest slightly outperforms Linear Regression, demonstrating effective model stability and reliability for deployment.
The Feature Engineering stage follows Exploratory Data Analysis, preparing the dataset for machine learning. It generates temporal and statistical features, encodes categorical identifiers, and ensures schema consistency. Implemented in AWS Glue, it enables reproducibility and scalability for model training, enhancing forecasting accuracy by incorporating lag and rolling average features.
This article expands on a previous post about building a serverless ETL pipeline on AWS by focusing on Exploratory Data Analysis (EDA). It details how to establish the EDA environment using AWS Glue and PySpark after cleaning the dataset. Key insights include sales trends, store and item performance, and correlation analysis, laying the groundwork for a demand forecasting model.
The post details enhancements made to a serverless ETL pipeline using AWS Glue and PySpark for retail sales data. Improvements include explicit column type conversions, missing value imputation, normalization of sales data, and integration of logging for observability. These changes aim to create a production-ready, machine-learning-friendly preprocessing layer for effective data analysis.
This project aims to develop a demand forecasting solution for retail using historical sales data from Kaggle. A data pipeline employing AWS Glue and PySpark will preprocess the data by cleaning and splitting it into training and testing sets. The objective is to maximize inventory management and customer satisfaction.
Last weekend, I explored GPU CUDA programming using AWS. Despite initial service quota issues, I successfully launched an EC2 instance equipped with an NVIDIA GPU. After setting up the environment, I compiled and ran a CUDA program, achieving a remarkable speedup of 151 times faster on the GPU compared to the CPU.
The author explores CUDA programming on AWS using an NVIDIA GPU, facing vCPU quota limitations that prevent launching an EC2 instance. After diagnosing the issue, they submitted a request for a quota increase through the Service Quotas console. The experience highlights the importance of checking AWS service limits when setting up GPU instances.
The article discusses the challenges of consistent application behavior in software development and how Docker addresses these issues. It outlines the creation of a simple Flask app, its containerization using Docker, and steps to ensure accessibility from outside the container. Troubleshooting and cleanup procedures are also covered, emphasizing a portable setup.
APIs are the backbone of modern applications. Among the different approaches to building them, one of the most widely used is the RESTful API. In…
Continue reading → Mastering RESTful API Basics for Developers
In this blog, we’ll build a simple distributed file service on AWS. The setup will have two file servers — Server 1 and Server 2.…
Continue reading → Setting Up a Simple Distributed File Service on AWS
Aviation data has always been fascinating. Planes crisscross the globe. Each one sends out tiny bursts of information as it soars through the sky. Thanks…
Hadoop’s MapReduce framework is a powerful tool for processing large-scale data in a distributed fashion. In this guide, we walk through setting up a Hadoop…
In today's interconnected world, real-time location tracking plays a crucial role in many industries. These include logistics and fleet management. It is also vital to…
Continue reading → Building a Real-Time GPS Data Processing System on AWS: A Step-by-Step Guide
The "SignatureDoesNotMatch" error often occurs when uploading files to AWS S3 due to signature mismatches related to secret keys. The author shares a step-by-step guide to troubleshoot this issue, which includes verifying IAM user credentials, configuring access keys, and successfully retrying the upload operation after resolving permissions.
Containers streamline application deployment by providing lightweight, isolated environments that ensure portability, scalability, and rapid deployment across systems. Unlike VMs, containers share the host OS kernel, reducing resource overhead while maintaining security and efficiency. Powered by Docker & Kubernetes, they enhance DevOps workflows, microservices architecture, and cloud computing. Ideal for fast, consistent deployments, containers eliminate compatibility issues, making them the go-to solution for modern software development. #Containers #Docker #Kubernetes #DevOps #CloudComputing
Cloud Computing: The Future of Scalable and Efficient IT Cloud computing is revolutionizing the way businesses and individuals store, process, and access data. By offering on-demand resources, seamless scalability, and cost efficiency, cloud technology eliminates the need for expensive on-premise hardware. Top industries, including AI, Big Data, and IoT, are leveraging cloud platforms like AWS, Google Cloud, and Microsoft Azure to drive innovation. With enhanced security, remote accessibility, and automated resource management, cloud computing is the future of IT infrastructure.
Virtualization in Cloud Computing: Key Concepts Virtualization enables efficient resource management in cloud computing by abstracting physical hardware into virtual machines (VMs). It enhances scalability, security, and system reliability while optimizing workloads. This summary covers hypervisors, VM migration, resource sharing, and hardware-assisted virtualization (Intel VT-x, AMD-V).
Recently I have been involved in doing to some source code compilation on an AWS EC2 instance. However, after the compilation I encountered a problem.…
Continue reading → Step-by-Step Guide: Filezilla Setup for AWS EC2
I am using an AWS EC2 instance for all my development works right now. And I am trying to figure out various ways to log…
Continue reading → How to SSH into AWS EC2 using MobaXterm
Once you have created an AWS account it has to be configured. The first thing you need to setup is the second level security called the MFA. This blog takes you through the step by step process of setting up MFA for your root user account.
A step by step guide for AWS beginners to setup their first AWS account to kick start with AWS cloud technology. It's free!
Tech For Talk 