In the new year, PyTorch has also received a major update. PyTorch 2.2 integrates new features such as FlashAttention-2 and AOTInductor, doubling the computing performance.
Following the release of version 2.1 at the PyTorch Conference last October, 521 developers from all over the world contributed 3,628 submissions, thus forming the latest PyTorch 2.2 version.
The new version integrates FlashAttention-2, making scaled_dot_product_attention (SDPA) has about 2 times performance improvement compared to the previous version.
PyTorch 2.2 also introduces a new TorchInductor ahead-of-time extension called AOTInductor, designed to compile and deploy PyTorch programs for non-python server-side.
in PyTorch torch.distributed A new abstraction called device_mesh is supported for initializing and representing ProcessGroups.

In addition, PyTorch 2.2 provides a standardized and configurable logging mechanism, TORCH_LOGS.
PyTorch 2.2 is still correct torch.compile Many improvements have been made, including improved support for compiler optimizers, as well as TorchInductor fusion and layout optimizations.

Finally, it’s worth noting that PyTorch will drop support for macOS x86. PyTorch 2.2.x This is the last version to support macOS x64.
What’s new in PyTorch 2.2
First, please note that if building PyTorch 2.2 from source requires GCC 9.4 or higher, the PyTorch codebase has been migrated from C++14 to C++17.

FlashAttention-2
FlashAttention-2 solves low-occupancy or unnecessary shared memory reads and writes by optimizing the work partitioning between different thread blocks and warps on the GPU.

FlashAttention-2 adjusts the algorithm to reduce the amount of non-matmul calculations, while improving the parallelism of Attention calculations (even a single head can span different thread blocks to increase occupancy). In each thread block, optimization Distribution of work between warps to reduce communication through shared memory.
PyTorch 2.2 updates the FlashAttention kernel to the v2 version. However, it should be noted that the previous Flash Attention kernel has a Windows implementation. Windows users can force the use of sdp_kernel and only enable the context manager of Flash Attention.

In 2.2, if you must use the sdp_kernel context manager, use the memory efficient or math kernel (on Windows).

With the support of FlashAttention-2,torch.nn.functional.scaled_dot_product_attention The speed is increased by about 2 times, reaching 50%-73% of the theoretical computing peak on the A100 GPU.
AOTInductor
AOTInductor is an extension of TorchInductor that processes exported PyTorch models, optimizes them, and generates shared libraries and other related artifacts.
These compiled artifacts can be deployed in non-Python environments and are often used for server-side inference.
The following example demonstrates how to call aot_compile to convert a model into a shared library.

AOTInductor supports the same backend as Inductor, including CUDA, ROCm, and CPU.
TORCH_LOGS
PyTorch 2.2 provides a standardized, configurable logging mechanism that can be used to analyze the status of various subsystems, such as compilation and distributed operations. Logging can be enabled through the TORCH_LOGS environment variable. For example, by modifying environment variables on the command line:

Set TorchDynamo's log level to logging.ERRORset the TorchInductor's log level to logging.DEBUG.
Of course, it can also be used in code as an API:

torch.distributed.device_mesh
PyTorch 2.2 introduces a new abstraction for representing ProcessGroups involved in distributed parallelism called torch.distributed.device_mesh.
Setting up a distributed communicator (NCCL) for distributed training can be cumbersome. Users need to write workloads with different degrees of parallelism and manually set up and manage the NCCL communicator (ProcessGroup) for each degree of parallelism.
This process can be complex and error-prone. DeviceMesh can simplify this process and make it easier to manage.
DeviceMesh is a higher-level abstraction for managing ProcessGroups. It allows users to effortlessly create inter-node and intra-node process groups without worrying about how to correctly set the ranks for different child process groups.
For example, one dimension of an array can represent data parallelism in FSDP, and another dimension can represent tensor parallelism in FSDP.
Users can also easily manage underlying process_groups through DeviceMesh to achieve multi-dimensional parallelism.

DeviceMesh is useful when dealing with multi-dimensional parallelism (such as 3D parallelism). As shown in the figure above, when your parallel solution requires communication across hosts and within each host, you can create a 2D grid that connects the devices in each host and connect each device with the Corresponding devices on other hosts are connected.
With the help of init_device_mesh (), we can complete the above 2D setup in just two lines:

And if we don't use DeviceMesh, we probably need to write the following code ourselves:

Of course, we can still access the underlying ProcessGroup if needed:

Optimizer improvements
There are probably the following points:
Compile optimizer improves performance in all benchmarks: HuggingFace +18%, TorchBench +19%, TIMM +8% E2E;
The compiled optimizer adds support for cudagraphs;
Averaging across all models in the test suite, the average benchmark compile time increases by approximately 40 seconds per test suite; ongoing optimizations may reduce this to under 30 seconds.
The main feature missing from the inductor for multi-tensor optimizer compilation is efficient encoding generation of the foreach operator.
Inside the scheduler, condense all buffer lists registered during devolution into ForeachKernelSchedulerNodes (a subclass of FusedSchedulerNode).
In order to check that the fusion is legal, the write operations performed by each internal SchedulerNode must match the read operations of the consuming SchedulerNode at the same list index.

Additionally, normal vertical fusion rules must allow fusion at every index of the consumer and producer SchedulerNode lists.
If these conditions are met, the ForeachKernelSchedulerNode will be vertically fused into one ForeachKernelSchedulerNode, where the corresponding point operations on each list will be fused.
By implementing this fusion, a series of foreach operations can be fused into a single core, allowing for complete fusion of multi-tensor optimizers.
Performance improvements
Many performance optimizations have been added to TorchInductor, including torch.concat Horizontal fusion support, improved convolution layout optimization, and improvements scaled_dot_product_attention Pattern matching.

PyTorch 2.2 also includes many performance enhancements for aarch64, including support for mkldnn weight prepacking, improved ideep primitive caching, and improved inference speed through fixed-format kernel improvements to OneDNN.
References:
https://pytorch.org/blog/pytorch2-2/