The study notes cover AI, ML, and deep learning, highlighting key concepts such as pre-trained and fine-tuned models, transfer learning, and their interpretability. It explores cognitive services across platforms like Azure and Google Cloud, emphasizing their applications in AI workflows, NLP, and computer vision tasks, along with practical examples and tools.
Table of Contents Distributed Computing (S1-25_CCZG526) Lab Assignment 1 Lab Assignment 2 Cloud Infrastructure and Systems Software (S1-25_CCZG502) Lab Assignment 1 API-driven Cloud Native Solutions…
Continue reading → BITS PILANI Assignments Second Semester M.Tech in Cloud Computing
The blog discusses the integration of Amazon EventBridge to automate AWS Glue workflows every two minutes, enhancing operational efficiency in data engineering and machine learning tasks. It details steps to create and configure EventBridge rules, set permissions, and verify workflows, emphasizing improvements in responsiveness, agility, and DataOps maturity.
The implemented stages of ingestion, preprocessing, EDA, and feature engineering have transitioned to automation and monitoring, forming a cohesive DataOps layer. By introducing orchestration, the independent Glue jobs become an automated, reliable workflow. Testing confirmed successful execution, paving the way for regular automations to enhance operations and insights from data.
The post discusses the evolution of a data pipeline, highlighting the integration of an API-driven layer for enhanced observability. This new functionality allows authorized users to access real-time operational status without manual checks across AWS services. The approach improves transparency, accountability, and agility while enabling proactive monitoring and automated responses in future enhancements.
This post details the development of a Machine Learning Pipeline for demand forecasting. Utilizing AWS Glue and PySpark, it covers training and evaluating Linear Regression and Random Forest models using an engineered feature dataset. Results show Random Forest slightly outperforms Linear Regression, demonstrating effective model stability and reliability for deployment.
The Feature Engineering stage follows Exploratory Data Analysis, preparing the dataset for machine learning. It generates temporal and statistical features, encodes categorical identifiers, and ensures schema consistency. Implemented in AWS Glue, it enables reproducibility and scalability for model training, enhancing forecasting accuracy by incorporating lag and rolling average features.
This article expands on a previous post about building a serverless ETL pipeline on AWS by focusing on Exploratory Data Analysis (EDA). It details how to establish the EDA environment using AWS Glue and PySpark after cleaning the dataset. Key insights include sales trends, store and item performance, and correlation analysis, laying the groundwork for a demand forecasting model.
The post details enhancements made to a serverless ETL pipeline using AWS Glue and PySpark for retail sales data. Improvements include explicit column type conversions, missing value imputation, normalization of sales data, and integration of logging for observability. These changes aim to create a production-ready, machine-learning-friendly preprocessing layer for effective data analysis.
This project aims to develop a demand forecasting solution for retail using historical sales data from Kaggle. A data pipeline employing AWS Glue and PySpark will preprocess the data by cleaning and splitting it into training and testing sets. The objective is to maximize inventory management and customer satisfaction.
Last weekend, I explored GPU CUDA programming using AWS. Despite initial service quota issues, I successfully launched an EC2 instance equipped with an NVIDIA GPU. After setting up the environment, I compiled and ran a CUDA program, achieving a remarkable speedup of 151 times faster on the GPU compared to the CPU.
The author explores CUDA programming on AWS using an NVIDIA GPU, facing vCPU quota limitations that prevent launching an EC2 instance. After diagnosing the issue, they submitted a request for a quota increase through the Service Quotas console. The experience highlights the importance of checking AWS service limits when setting up GPU instances.
The PDF outlines the evolution of computer generations, highlighting key advancements from vacuum tubes to quantum computing. It covers various architectures, memory systems, and performance concepts, emphasizing the impact of Moore's Law. Additionally, it discusses embedded systems, operating systems roles, and provides case studies on RAM speeds and server requirements for modern workloads.
Lesson 1 & Part 2 Introduction to Distributed Computing Types of Distributed Systems Flynn’s Taxonomy Remote Procedure Call (RPC) Message Queueing Publish–Subscribe Pattern Logical Clocks…
Continue reading → Distributed Computing Notes
The provided link directs to a PDF document containing answers to Sample Questions Set 1. Users can access the resource for educational purposes, likely to aid in understanding specific topics or prepare for assessments. The content serves as a study aid for individuals seeking clarification on the questions presented.
Lessons Lesson 6: Docker Container https://techfortalk.co.uk/wp-content/uploads/2025/09/devops-lesson-6_-docker-container.pdf Virtualization Notes https://techfortalk.co.uk/wp-content/uploads/2025/09/virtualisation.pdf GIT Notes (Lesson 4&5) https://techfortalk.co.uk/wp-content/uploads/2025/09/devops-lesson-45-git.pdf Questions & Answers https://techfortalk.co.uk/wp-content/uploads/2025/09/devops-midsem-questions.pdf Past Paper Q&A https://techfortalk.co.uk/wp-content/uploads/2025/10/devops-past-paper-qa-1.pdf
The article discusses the challenges of consistent application behavior in software development and how Docker addresses these issues. It outlines the creation of a simple Flask app, its containerization using Docker, and steps to ensure accessibility from outside the container. Troubleshooting and cleanup procedures are also covered, emphasizing a portable setup.
APIs are the backbone of modern applications. Among the different approaches to building them, one of the most widely used is the RESTful API. In…
Continue reading → Mastering RESTful API Basics for Developers
In this blog, we’ll build a simple distributed file service on AWS. The setup will have two file servers — Server 1 and Server 2.…
Continue reading → Setting Up a Simple Distributed File Service on AWS
Acronyms, like RAII (Resource Acquisition Is Initialization), can be intimidating for programmers but reveal their elegance once understood. RAII ties resource management to object lifetime, ensuring reliable cleanup even during exceptions. This blog illustrates its significance through examples, emphasizing its role in modern C++ and urging developers to adopt its principles.
Aviation data has always been fascinating. Planes crisscross the globe. Each one sends out tiny bursts of information as it soars through the sky. Thanks…
Hadoop’s MapReduce framework is a powerful tool for processing large-scale data in a distributed fashion. In this guide, we walk through setting up a Hadoop…
The blog discusses a method for multiplying a large square matrix by a vector using MPI with a block-column distribution strategy. It describes how Process 0 distributes matrix columns to different processes, which calculate local products and then combine results using MPI Reduce scatter. An understanding of vector multiplication is emphasized, explaining how vectors are represented in C, including examples of single and square vectors. The process of matrix-vector multiplication is detailed with a C code snippet, demonstrating each multiplication step and the final result. The blog prepares readers for implementing parallel computations in MPI, enhancing efficiency.
In today's interconnected world, real-time location tracking plays a crucial role in many industries. These include logistics and fleet management. It is also vital to…
Continue reading → Building a Real-Time GPS Data Processing System on AWS: A Step-by-Step Guide
This content discusses the challenges of measuring message-passing performance in a distributed system, specifically using a ping-pong pattern with MPI. It highlights the limitations of the C clock() function for timing short exchanges, as it may return zero or inconsistent results when few iterations occur. To obtain reliable data, the post recommends a dynamic iteration scaling approach—starting with a small number of iterations and doubling it until a measurable time is recorded. This method ensures accurate measurements across varying hardware and system loads, ultimately providing a robust benchmark for MPI communication costs essential for optimization in high-performance computing.
The post discusses an advanced problem in distributed computing using MPI (Message Passing Interface) for a large e-commerce operation. It focuses on collecting local minimum and maximum shipping times from various global warehouse hubs to calculate overall global shipping times. The program simulates generating these times using C's random number generator, ensuring the correct relationship between min and max. It applies MPI_Reduce() to aggregate results efficiently across nodes. The author encourages experimentation with different randomization methods and varying the number of MPI processes while providing a GitHub repository for further exploration of relevant MPI examples.
In distributed computing, MPI programming utilizes a root node to manage data distribution and result aggregation among multiple nodes. The MPI_Reduce() function plays a critical role in performing global computations efficiently, allowing nodes to send data and gather results via message passing. Each non-root node computes its contributions, while the root node consolidates them. The function requires parameters such as sendbuf, recvbuf, count, datatype, op, root, and comm to operate effectively. While MPI_Reduce() returns results only to the root, MPI_Allreduce() disseminates results across all nodes. This understanding of MPI_Reduce() lays the groundwork for complex computational challenges.
In the described scenario, the online shopping platform "QuickBuy" offers a limited-time discount for a product, allowing only 10 customers to buy at a reduced price. The developer, Alex, uses multithreading with Pthreads to manage simultaneous purchase attempts. Mutex locks ensure that no more than 10 customers can modify the shared stock resource at the same time, preventing race conditions. The program simulates customer threads that compete for the limited inventory while employing condition variables to manage wait states. The main function oversees the sale's timing and ensures that excess customers are informed when the offer ends.
This blog post discusses how multi-tasking enables efficient CPU time-sharing among programs, allowing them to seemingly run simultaneously on a single-core processor. The OS scheduler manages task switching, allowing programs like a music player and a word processor to share CPU time effectively. Context switching is a rapid process that gives the appearance of parallel execution. However, distinct processes have isolated memory spaces, complicating data sharing. Threads within a process, on the other hand, share address space, simplifying communication and resource management. This post also introduces the pthread library for creating threads in C, showcasing the practicality of multi-threading.
Question 4 You are tasked with developing a multi-threaded system using Pthreads to manage a dynamic task queue. The program should start with a user-specified…
Continue reading → Building a Multi-Threaded Task Queue with Pthreads
The content explains that a uni-processor system has only one CPU, which can execute only one piece of code at a time. This leads to pseudo parallelism, where multiple programs seem to run simultaneously by sharing CPU time. For illustration, two simple programs are presented: one continuously prints "Hello World" and the other prints "Hello Boss." In practice, they take turns using the CPU, facilitated by the operating system's scheduler. The blog emphasizes terminologies like process and infinite loop, providing insights into how parallelism works, even in environments with limited processing capabilities.
This document introduces Big Data and its challenges, highlighting Hadoop as a scalable solution for distributed storage and parallel processing. It explains HDFS (Hadoop Distributed File System) for fault-tolerant storage, MapReduce for distributed computing, and YARN for resource management. Hadoop follows a Master-Slave architecture, where the Master Node (JobTracker, NameNode) assigns tasks, and Slave Nodes (TaskTrackers, DataNodes) process data. The document details the MapReduce workflow, from mapping, sorting, shuffling, and reducing. Real-world applications, including its adoption by Facebook, Amazon, and IBM, are discussed. It also touches on Hadoop deployment on AWS EMR for cloud-based big data processing.
Hadoop is a powerful framework that enables distributed processing of large datasets. It follows the MapReduce paradigm. Computation is broken down into independent map and…
Continue reading → Set Up a Hadoop Cluster on AWS EMR: A Step-by-Step Guide
The "SignatureDoesNotMatch" error often occurs when uploading files to AWS S3 due to signature mismatches related to secret keys. The author shares a step-by-step guide to troubleshoot this issue, which includes verifying IAM user credentials, configuring access keys, and successfully retrying the upload operation after resolving permissions.
Containers streamline application deployment by providing lightweight, isolated environments that ensure portability, scalability, and rapid deployment across systems. Unlike VMs, containers share the host OS kernel, reducing resource overhead while maintaining security and efficiency. Powered by Docker & Kubernetes, they enhance DevOps workflows, microservices architecture, and cloud computing. Ideal for fast, consistent deployments, containers eliminate compatibility issues, making them the go-to solution for modern software development. #Containers #Docker #Kubernetes #DevOps #CloudComputing
Cloud Computing: The Future of Scalable and Efficient IT Cloud computing is revolutionizing the way businesses and individuals store, process, and access data. By offering on-demand resources, seamless scalability, and cost efficiency, cloud technology eliminates the need for expensive on-premise hardware. Top industries, including AI, Big Data, and IoT, are leveraging cloud platforms like AWS, Google Cloud, and Microsoft Azure to drive innovation. With enhanced security, remote accessibility, and automated resource management, cloud computing is the future of IT infrastructure.
Routing Information Protocol next generation (RIPng) is a distance-vector routing protocol tailored for IPv6 networks. Built on the Bellman-Ford algorithm, it serves as an Interior Gateway Protocol (IGP) for moderate-sized autonomous systems, leveraging multicast for routing updates and a hop-count metric with a 15-hop limit. RIPng enhances IPv6 routing efficiency but inherits limitations such as slow convergence and the "counting to infinity" problem. Despite its simplicity and ease of deployment, it lacks dynamic route cost adjustments, making it less suitable for complex or large-scale networks. This document provides an in-depth exploration of RIPng’s protocol structure, routing table mechanics, and message formats, offering insights into its practical applications and constraints in IPv6 networking.
Virtualization in Cloud Computing: Key Concepts Virtualization enables efficient resource management in cloud computing by abstracting physical hardware into virtual machines (VMs). It enhances scalability, security, and system reliability while optimizing workloads. This summary covers hypervisors, VM migration, resource sharing, and hardware-assisted virtualization (Intel VT-x, AMD-V).
Recently I have been involved in doing to some source code compilation on an AWS EC2 instance. However, after the compilation I encountered a problem.…
Continue reading → Step-by-Step Guide: Filezilla Setup for AWS EC2
I am using an AWS EC2 instance for all my development works right now. And I am trying to figure out various ways to log…
Continue reading → How to SSH into AWS EC2 using MobaXterm
Introduction to Parallel Computing: Why It’s the Future of High-Performance Computing Parallel computing is transforming high-performance computing, AI, big data, and scientific simulations by distributing tasks across multiple processors for faster execution. This guide covers what parallel computing is, why it matters in fields like climate modelling and real-time analytics, and how parallel programming models like MPI, OpenMP, and CUDA optimize workloads. Ready to explore parallel computing? Read the full article to see how multi-core and GPU-based computing power the future of technology.
Introduction to Distributed Memory Programming with MPI Parallel computing is essential for high-performance applications, and the Message-Passing Interface (MPI) is a key framework for distributed memory programming. Unlike shared-memory systems, distributed-memory architectures require explicit communication between processes, enabling scalable computing across clusters. This article explores MPI fundamentals, including message-passing, process management, and parallel computation techniques. Key topics covered include: Blocking vs. Non-blocking communication for efficient data transfer Collective operations like MPI_Bcast, MPI_Gather, and MPI_Reduce Process synchronization using MPI_Barrier Vector partitioning strategies (Block, Cyclic, and Block-Cyclic) for load balancing Performance evaluation and scalability in parallel applications Whether you're new to MPI or looking to optimize distributed computing tasks, this guide will help you implement efficient parallelism in C and Fortran using MPI.
Efficient parallel processing relies on smart data partitioning strategies to distribute workloads across multiple processors. This blog explores three fundamental techniques: Block Partitioning, Cyclic Partitioning, and Block-Cyclic Partitioning. Through step-by-step explanations and visual diagrams, you'll learn how these methods optimize performance by balancing data distribution. Whether you're new to parallel computing or looking to refine your understanding, this guide breaks down complex concepts into simple, digestible insights. Stay tuned for a practical implementation using MPI (Message Passing Interface) in the next section!
Upgrade your STM32MP1's WiFi driver effortlessly with this comprehensive guide. Learn how to replace the outdated driver with an updated NXP version using Buildroot. Discover step-by-step instructions for obtaining the NXP WiFi driver source code, configuring it for your target, and integrating it into Buildroot. With clear explanations and helpful tips, this blog post simplifies the process, ensuring smooth implementation. Get ready to enjoy improved performance with your newly compiled NXP WiFi kernel module!
Recently, I found myself working with a cellular USB dongle on an embedded Linux system. I've previously documented my experience integrating an LTE device, the…
Continue reading → Setting Up Data Connection with SORACOM Onyx LTE USB Modem on Linux
In this chapter, we will learn various aspects of class template instantiation. Understanding the class template instantiation is pivotal to the overall understanding of the functioning of templates in general. The fundamental concept of template instantiation remains the same for both function templates and class templates. However, instantiation of class template instantiation is more complex than function template.
The content outlines seven chapters on C++ templates, covering introductory concepts, function templates, template parameters, return types, miscellaneous aspects, class templates, and class template instantiation. Each chapter provides a foundation for understanding and effectively using templates in C++. Additionally, it mentions the availability of source code.
Tech For Talk 