本书内容简洁、直观、实用，强调计算思维能力和并行编程技巧。本书主要分为四个部分：第 一部分介绍异构并行计算编程的基础概念，包括数据并行化、GPU架构、CUDA编程及程序性能优化方法等内容；第二部分介绍并行模式，包括卷积、模板、并行直方图、归约、前缀和、归并等内容；第三部分介绍高级模式及应用，包括排序、稀疏矩阵计算、图遍历、深度学习、迭代式磁共振成像重建、静电势能图和计算思维等内容；第四部分介绍高级编程实践，包括异构计算集群编程、CUDA动态并行化等内容。本书不仅适合高等院校计算机相关专业的学生学习，也适合并行计算领域的技术人员参考。

胡文美（Wen-mei W. Hwu） NVIDIA公司杰出研究科学家兼高级研究总监。伊利诺伊大学厄巴纳-香槟分校荣休教授，并行计算研究中心首席科学家。他在编译器设计、计算机体系结构、微体系结构和并行计算方面做出了卓越贡献，是IEEE Fellow、ACM Fellow，荣获了包括ACM-IEEE CS Eckert-Mauchly奖、ACM Grace Murray Hopper奖、ACM SIGARCH Maurice Wilkes奖在内的众多奖项。他拥有加州大学伯克利分校计算机科学博士学位。

大卫·B. 柯克（David B. Kirk） 美国国家工程院院士，NVIDIA Fellow，曾任NVIDIA公司首席科学家。2002年，他荣获ACM SIGGRAPH计算机图形成就奖，以表彰其在把高性能计算机图形系统推向大众市场方面做出的杰出贡献。他拥有加州理工学院计算机科学博士学位。

伊扎特·埃尔·哈吉（Izzat El Hajj） 贝鲁特美国大学计算机科学系助理教授。他的研究方向是针对新兴并行处理器和内存技术的应用加速和编程支持，特别是GPU和内存内处理。他拥有伊利诺伊大学厄巴纳-香槟分校电气与计算机工程博士学位。

Contents <br /><br />Foreword <br />Preface　<br />Acknowledgments　 <br /><br />CHAPTER 1 Introduction 1 <br /><br />1.1 Heterogeneous parallel computing 3 <br /><br />1.2 Why more speed or parallelism? 7 <br /><br />1.3 Speeding up real applications 9 <br /><br />1.4 Challenges in parallel programming 11 <br /><br />1.5 Related parallel programming interfaces 13 <br /><br />1.6 Overarching goals 14 <br /><br />1.7 Organization of the book 15 <br />References 19 <br /><br />Part I Fundamental Concepts <br /><br />CHAPTER 2 Heterogeneous data parallel computing　23<br />With special contribution from David Luebke <br /><br />2.1 Data parallelism 23 <br /><br />2.2 CUDA C program structure 27 <br /><br />2.3 A vector addition kernel 28 <br /><br />2.4 Device global memory and data transfer 31 <br /><br />2.5 Kernel functions and threading 35 <br /><br />2.6 Calling kernel functions 40 <br /><br />2.7 Compilation 42 <br /><br />2.8 Summary 43 <br />Exercises 44 <br />References 46 <br /><br />CHAPTER 3 Multidimensional grids and data 47 <br /><br />3.1 Multidimensional grid organization 47 <br /><br />3.2 Mapping threads to multidimensional data 51 <br /><br />3.3 Image blur: a more complex kernel 58 <br /><br />3.4 Matrix multiplication 62 <br /><br />3.5 Summary 66 <br />Exercises 67 <br /><br />CHAPTER 4 Compute architecture and scheduling 69 <br /><br />4.1 Architecture of a modern GPU 70 <br /><br />4.2 Block scheduling 70 <br /><br />4.3 Synchronization and transparent scalability 71 <br /><br />4.4 Warps and SIMD hardware 74 <br /><br />4.5 Control divergence 79 <br /><br />4.6 Warp scheduling and latency tolerance 83 <br /><br />4.7 Resource partitioning and occupancy 85 <br /><br />4.8 Querying device properties 87 <br /><br />4.9 Summary 90 <br />Exercises 90 <br />References 92 <br /><br />CHAPTER 5 Memory architecture and data locality 93 <br /><br />5.1 Importance of memory access efficiency 94 <br /><br />5.2 CUDA memory types 96 <br /><br />5.3 Tiling for reduced memory traffic 103 <br /><br />5.4 A tiled matrix multiplication kernel 107 <br /><br />5.5 Boundary checks 112 <br /><br />5.6 Impact of memory usage on occupancy 115 <br /><br />5.7 Summary 118 <br />Exercises　119 <br /><br />CHAPTER 6 Performance considerations 123 <br /><br />6.1 Memory coalescing 124 <br /><br />6.2 Hiding memory latency 133 <br /><br />6.3 Thread coarsening 138 <br /><br />6.4 A checklist of optimizations 141 <br /><br />6.5 Knowing your computation’s bottleneck 145 <br /><br />6.6 Summary 146 <br />Exercises　146 <br />References 147 <br /><br />Part II Parallel Patterns <br /><br />CHAPTER 7 Convolution <br />An introduction to constant memory and caching 151 <br /><br />7.1 Background 152 <br /><br />7.2 Parallel convolution: a basic algorithm 156 <br /><br />7.3 Constant memory and caching 159 <br /><br />7.4 Tiled convolution with halo cells 163 <br /><br />7.5 Tiled convolution using caches for halo cells 168 <br /><br />7.6 Summary 170 <br />Exercises　171 <br /><br />CHAPTER 8 Stencil 173 <br /><br />8.1 Background 174 <br /><br />8.2 Parallel stencil: a basic algorithm 178 <br /><br />8.3 Shared memory tiling for stencil sweep 179 <br /><br />8.4 Thread coarsening 183 <br /><br />8.5 Register tiling 186 <br /><br />8.6 Summary 188 <br />Exercises　188 <br /><br />CHAPTER 9 Parallel histogram 191 <br /><br />9.1 Background 192 <br /><br />9.2 Atomic operations and a basic histogram kernel 194 <br /><br />9.3 Latency and throughput of atomic operations 198 <br /><br />9.4 Privatization 200 <br /><br />9.5 Coarsening 203 <br /><br />9.6 Aggregation 206 <br /><br />9.7 Summary 208 <br />Exercises　209 <br />References 210 <br /><br />CHAPTER 10 Reduction <br />And minimizing divergence 211 <br /><br />10.1 Background 211 <br /><br />10.2 Reduction trees 213 <br /><br />10.3 A simple reduction kernel 217 <br /><br />10.4 Minimizing control divergence 219 <br /><br />10.5 Minimizing memory divergence 223 <br /><br />10.6 Minimizing global memory accesses 225 <br /><br />10.7 Hierarchical reduction for arbitrary input length 226 <br /><br />10.8 Thread coarsening for reduced overhead 228 <br /><br />10.9 Summary 231 <br />Exercises　232 <br /><br />CHAPTER 11 Prefix sum (scan) <br />An introduction to work efficiency in parallel algorithms 235 <br /><br />With special contributions from Li-Wen Chang, Juan Go′mez-Luna and John Owens <br /><br />11.1 Background 236 <br /><br />11.2 Parallel scan with the Kogge-Stone algorithm 238 <br /><br />11.3 Speed and work efficiency consideration 244 <br /><br />11.4 Parallel scan with the Brent-Kung algorithm 246 <br /><br />11.5 Coarsening for even more work efficiency 251 <br /><br />11.6 Segmented parallel scan for arbitrary-length inputs 253 <br /><br />11.7 Single-pass scan for memory access efficiency 256 <br /><br />11.8 Summary 259 <br />Exercises　260 <br />References 261 <br /><br />CHAPTER 12 Merge <br />An introduction to dynamic input data identification 263 <br /><br />With special contributions from Li-Wen Chang and Jie Lv <br /><br />12.1 Background 263 <br /><br />12.2 A sequential merge algorithm 265 <br /><br />12.3 A parallelization approach 266 <br /><br />12.4 Co-rank function implementation 268 <br /><br />12.5 A basic parallel merge kernel 273 <br /><br />12.6 A tiled merge kernel to improve coalescing 275 <br /><br />12.7 A circular buffer merge kernel 282 <br /><br />12.8 Thread coarsening for merge 288 <br /><br />12.9 Summary 288 <br />Exercises　289 <br />References 289 <br /><br /><br />Part III Advanced Patterns and Applications <br /><br />CHAPTER 13 Sorting 293 <br />With special contributions from Michael Garland <br /><br />13.1 Background 294 <br /><br />13.2 Radix sort 295 <br /><br />13.3 Parallel radix sort 296 <br /><br />13.4 Optimizing for memory coalescing 300 <br /><br />13.5 Choice of radix value 302 <br /><br />13.6 Thread coarsening to improve coalescing 305 <br /><br />13.7 Parallel merge sort 306 <br /><br />13.8 Other parallel sort methods 308 <br /><br />13.9 Summary 309 <br />Exercises　310 <br />References 310 <br /><br />CHAPTER 14 Sparse matrix computation 311 <br /><br />14.1 Background 312 <br /><br />14.2 A simple SpMV kernel with the COO format 314 <br /><br />14.3 Grouping row nonzeros with the CSR format 317 <br /><br />14.4 Improving memory coalescing with the ELL format 320 <br /><br />14.5 Regulating padding with the hybrid ELL-COO format 324 <br /><br />14.6 Reducing control divergence with the JDS format 325 <br /><br />14.7 Summary 328 <br />Exercises　329 <br />References 329 <br /><br />CHAPTER 15 Graph traversal 331 <br />With special contributions from John Owens and Juan Go′mez-Luna <br /><br />15.1 Background 332 <br /><br />15.2 Breadth-first search 335 <br /><br />15.3 Vertex-centric parallelization of breadth-first search 338 <br /><br />15.4 Edge-centric parallelization of breadth-first search 343 <br /><br />15.5 Improving efficiency with frontiers.345 <br /><br />15.6 Reducing contention with privatization 348 <br /><br />15.7 Other optimizations 350 <br /><br />15.8 Summary 352 <br />Exercises　353 <br />References 354 <br /><br />CHAPTER 16 Deep learning 355 <br />With special contributions from Carl Pearson and Boris Ginsburg<br /> <br />16.1 Background 356 <br /><br />16.2 Convolutional neural networks 366 <br /><br />16.3 Convolutional layer: a CUDA inference kernel 376 <br /><br />16.4 Formulating a convolutional layer as GEMM 379 <br /><br />16.5 CUDNN library 385 <br /><br />16.6 Summary 387 <br />Exercises　388 <br />References 388 <br /><br />CHAPTER 17 Iterative magnetic resonance imaging reconstruction 391 <br />17.1 Background 391 <br /><br />17.2 Iterative reconstruction 394 <br /><br />17.3 Computing FHD 396 <br /><br />17.4 Summary 412 <br />Exercises　413 <br />References 414 <br /><br />CHAPTER 18 Electrostatic potential map 415 <br />With special contributions from John Stone <br /><br />18.1 Background 415 <br /><br />18.2 Scatter versus gather in kernel design 417 <br /><br />18.3 Thread coarsening 422 <br /><br />18.4 Memory coalescing 424 <br /><br />18.5 Cutoff binning for data size scalability 425 <br /><br />18.6 Summary.430 <br />Exercises　431 <br />References 431 <br /><br />CHAPTER 19 Parallel programming and computational thinking 433 <br /><br />19.1 Goals of parallel computing 433 <br /><br />19.2 Algorithm selection 436 <br /><br />19.3 Problem decomposition 440 <br /><br />19.4 Computational thinking 444 <br /><br />19.5 Summary 446 <br />References 446 <br /><br /><br />Part IV Advanced Practices <br /><br />CHAPTER 20 Programming a heterogeneous computing cluster <br />An introduction to CUDA streams 449　<br /><br />With special contributions from Isaac Gelado and Javier Cabezas　<br /><br />20.1　Background 449　<br />20.2　A running example 450　<br />20.3　Message passing interface basics 452　<br />20.4　Message passing interface point-to-point communication 455　<br />20.5　Overlapping computation and communication 462　<br />20.6　Message passing interface collective communication 470　<br />20.7　CUDA aware message passing interface 471　<br />20.8　Summary 472　<br />Exercises　472　<br />References 473　<br /><br />CHAPTER 21 CUDA dynamic parallelism 475 <br />With special contributions from Juan Go′mez-Luna <br /><br />21.1 Background 476 <br /><br />21.2 Dynamic parallelism overview 478 <br /><br />21.3 An example: Bezier curves 481 <br /><br />21.4 A recursive example: quadtrees 484 <br /><br />21.5 Important considerations 490 <br /><br />21.6 Summary 492 <br />Exercises　493 <br />A21.1 Support code for quadtree example 495 <br />References 497 <br /><br />CHAPTER 22 Advanced practices and future evolution 499 <br /><br />With special contributions from Isaac Gelado and Mark Harris <br />22.1 Model of host/device interaction 500 <br /><br />22.2 Kernel execution control 505 <br /><br />22.3 Memory bandwidth and compute throughput 508 <br /><br />22.4 Programming environment 510 <br /><br />22.5 Future outlook 513 <br />References 513 <br /><br />CHAPTER 23 Conclusion and outlook 515 <br /><br />23.1 Goals revisited 515 <br /><br />23.2 Future outlook 516 <br /><br />Appendix A: Numerical considerations 519 <br />Index 537