Parallel Programming
- Introduction to Parallel Programming
- Need for parallelism
- Limits of serial computation (Moore’s Law)
- Data parallelism vs task parallelism
- Amdahl’s Law and Gustafson’s Law
- Hardware and Architectures
- Multicore CPUs and manycore GPUs
- SIMD vs MIMD architectures
- Shared memory vs distributed memory
- Interconnects and topologies (bus, ring, mesh, torus)
- Programming Models and Paradigms
- Shared memory programming (threads, OpenMP)
- Distributed memory programming (MPI, RPC)
- Dataflow programming
- Actor model
- Parallel Programming Languages and APIs
- OpenMP (C/C++)
- MPI (C/C++, Fortran)
- CUDA (NVIDIA GPUs)
- OpenCL / SYCL
- Python multiprocessing & concurrent.futures
- Synchronization and Communication
- Race conditions and critical sections
- Mutexes, semaphores, barriers
- Deadlocks, livelocks, starvation
- Message passing and collective communication
- Parallel Algorithms and Patterns
- Divide and conquer
- Fork-join
- MapReduce
- Pipeline and task parallelism
- Producer-consumer
- Parallel sorting algorithms
- Matrix operations and prefix sums
- Performance Analysis and Optimization
- Speedup, efficiency, and scalability
- Load balancing strategies
- Memory hierarchy and cache performance
- Avoiding false sharing
- Profiling tools (e.g., gprof, perf, Intel VTune)
- Debugging and Testing Parallel Code
- Common concurrency bugs
- Tools for race detection and debugging (Valgrind, Helgrind, Intel Inspector)
- Reproducibility and testing strategies
- GPU and Heterogeneous Programming
- GPU architecture overview
- CUDA memory model and kernels
- Streams and concurrency
- Host-device synchronization
- OpenCL for cross-platform GPU/CPU computing
- Cloud and Distributed Computing
- Hadoop and Spark
- Serverless and microservice architectures
- Distributed task scheduling
- Actor-based frameworks (e.g., Akka)
- Applications of Parallel Programming
- Scientific computing
- Real-time rendering and graphics
- Machine learning and deep learning
- Financial modeling and simulations
- Emerging and Advanced Topics
- Parallelism in quantum computing
- Neuromorphic and brain-inspired computing
- Parallel ML frameworks (TensorFlow, PyTorch)
- Parallel databases and data processing systems