Course Highlights:

This course is a foundational two-day course (with an optional third day) that offers concepts and techniques for parallel programming. Going beyond treating parallel programming as yet another programming language to be learned, this course will cover aspects of parallel architectures, parallel algorithms, concurrency and correctness issues. It will then dive into the two main parallel programming paradigms, namely shared memory and message passing. The students will also be taught to think about different forms of parallelism and how it can fit their application domains. The main emphasis of the course is to get experienced software professionals, and code developers to “think parallel”, as well as equip them with the skills to translate that thinking into actual implementation on modern platforms. The course will have a good balance of theory and practice. The bulk of the “hands-on” sessions in the main two day course will be based on the industry-strength OpenMP and MPI standards. For the optional third day, massively parallel general purpose programming on graphics processors using CUDA, and cloud computing using Google’s MapReduce will be covered.

Objective of the course:

Hardware has always been designed to run in parallel. Software, except for the operating systems, has largely been sequential because sequential code is easier to reason with. However, advances in hardware technology are now such that sequential software will simply not scale. While in the past, shared memory multiprocessors were confined to large servers, today one can easily buy a off-the-shelf CPU with 4 or more “cores” inside. In other words, to exploit the best that modern processors can offer will require the software developer to think and implement parallelism. However, the migration from programming sequentially to parallel programming is more than just learning a new programming language. Parallel programming involves new paradigms of partitioning the problem, and expressing the solution. In this course, we shall learn about:

• Why the need for parallel software development?
• How is parallel programming different from “normal” programming?
• What are the ways parallelism is expressed?
• How can I transform and scale my sequential application to a modern parallel platform?

This course is particularly hands-on. It is not a “theory” course although it will give the audience the necessary foundation for understanding and thinking in parallel. The emphasis is on real code, real platform, and real applications.

Who Should Attend:

Code developers or other professionals who have experience in C/C++ programming. The course does not assume prior knowledge in multicore architecture or parallel programming. In order to acquire a complete skill set on multicore programming, participants are encouraged to register for a follow up course “Advanced Programming on Multicore: Writing and Tuning for Performance and Scalability”, which focuses on how to structure and tune programs for maximum performance and scalability to a large number of cores.

Course Outline:

• Parallel architectures
• A survey of modern computer architectures
• The need for parallel programming
• Trends and implications for software development

• Parallel programming models and languages
• Flynn’s classification
• A survey of parallel programming paradigms
• Shared memory models
• Distributed memory models
• Grids and clouds

• Parallel algorithms
• Performance metrics
• Amdahl’s Law and Scaling
• Types of parallel algorithms
• Sorting and searching in parallel
• Matrix algorithms
• Graph algorithms
• Patterns of parallelism

• Concurrency, correctness and optimization
• Sharing and contention
• Synchronization and locking
• Deadlocks
• Livelocks
• Race conditions
• Termination
• Partitioning, mapping and associated overhead
• Granularity of locks
• Scheduling and Load Balancing
• Cache issues

• Shared memory programming (I) – Thread programming
• Introduction to Pthreads
• Locks and semaphores in Pthreads
• Conditional variables
• Barriers
• Pipeline parallelism
• Functional parallelism
• Double buffering
• Events and interrupts
• Case studies

• Shared memory programming (II) – OpenMP
• Loop level parallelism
• Loop level dependences
• Introduction to OpenMP
• Privatization
• Reduction
• Scheduling and load balancing in OpenMP
• Limitations of OpenMP
• Case studies

• Message passing programming – MPI
• Distributed memory paradigm
• Introduction to MPI
• Communication and synchronization
• Load balancing
• Case studies

• (Optional) CUDA programming
• Introduction to GPGPUs
• Introduction to CUDA programming model
• Case studies

• (Optional) MapReduce

• Introduction to Grid and Cloud computing
• Introduction to MapReduce
• Case studies