Learn CUDA Programming by Jaegeun Han

Author:Jaegeun Han
Language: eng
Format: epub, mobi
Tags: COM051070 - COMPUTERS / Programming Languages / C++, COM048000 - COMPUTERS / Systems Architecture / Distributed Systems and Computing, COM051220 - COMPUTERS / Programming / Parallel
Publisher: Packt Publishing
Published: 2019-09-27T12:32:24+00:00

Scalable Multi-GPU Programming

So far, we have concentrated on getting optimal performance on a single GPU. Dense nodes with multiple GPUs have become a pressing need for upcoming supercomputers, especially since the ExaFLOP (a quintillion operations per sec) system is becoming a reality. GPU architecture is energy-efficient and hence, in recent years, systems with GPUs have taken the majority of the top spots in the Green500 list (https://www.top500.org/green500). In Green500's November 2018 list, seven out of the top 10 systems were based on the NVIDIA GPU.

The DGX system from NVIDIA now has 16 V100 32 GB in one server. With the help of unified memory and interconnect technologies such as NVLink and NvSwitch, developers can see all GPUs as one big GPU with 512 GB memory (16 GPU * 32 GB each). In this chapter, we will go into the details of writing CUDA code and make use of CUDA-aware libraries to efficiently get scalability in a multi-GPU environment within and across nodes.

In this chapter, we will cover the following topics:

Solving a linear equation using Gaussian elimination

GPUDirect peer to peer

A brief introduction to MPI

GPUDirect RDMA

CUDA streams

Additional tricks

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.