The COMET compiler consists of a Domain Specific Language (DSL) for sparse and dense tensor algebra computations, a progressive lowering process to map high-level operations to low-level architectural resources, a series of optimizations performed in the lowering process, and various IR dialects to represent key concepts, operations, and types at each level of the multi-level IR. At each level of the IR stack, COMET performs different optimizations and code transformations. Domain-specific, hardware- agnostic optimizations that rely on high-level semantic information are applied at high-level IRs. These include reformulation of high-level operations in a form that is amenable for execution on heterogeneous devices (e.g., rewriting Tensor contraction operations as Transpose-Transpose-GEMM-Transpose) and automatic parallelization of high-level primitives (e.g., tiling for thread- and task-level parallelism).
Comprehensive documentation of the COMET compiler can be found here.
These commands can be used to setup COMET project:
- Requirements. To install COMET and LLVM/MLIR, the following dependencies need to be already installed:
- CMake (3.25 or later)
- Ninja (1.5 or later)
- C++ compiler toolchain as mentioned here
- Python3 (3.9 or later)
- Git (1.8.4 or later)
- pkg-config (0.29.2 or later)
When targeting GPUs or/and FPGAs you will also need the drivers and runtimes of the respective vendors (Nvidia/CUDA, AMD/ROCm, Xilinx/XRT,Vitis).
1.a [Optional but recommended] Create a new python environment
$ export PYTHON_EXECUTABLE=$(which python3.x) # Replace 3.x with your version
$ ${PYTHON_EXECUTABLE} -m venv "comet"
$ source comet/bin/activate- Build COMET.
LLVM and blis are dependencies included in this repo as git submodules that point to the respective versions of the libraries that COMET has been tested with. LLVM/MLIR are changing relatively rapidly, so feel free to use the current version of LLVM, but APIs may have changed.
BLIS is an award-winning portable software framework for instantiating high-performance BLAS-like dense linear algebra libraries. COMET generates a call to BLIS microkernel after some optimizations. Also, blis is patched with changes specific to COMET, so an existing installation may not be used.
To build COMET for CPU execution only, run the following commands:
$ mkdir build
$ cd build
$ cmake ../
$ makeThis will fetch and build the LLVM and blis dependencies automatically, and build COMET. Once the command completes COMET will be installed in build/comet/.
You may also specify a custom LLVM installation, instead of downloading a fresh copy, by passing its path to the cmake command:
$ cmake ../ -DLLVM_CUSTOM_BUILD_PATH=/path/to/llvm/build/Note: The LLVM installation should have enabled the mlir, openmp, clang projects.
Once complete, you can run the integration tests using the following commands:
$ cd comet
$ ninja check-comet-integrationThe -DCMAKE_BUILD_TYPE=DEBUG flag enables debug information, which makes the
whole tree compile slower, but allows you to step through code into COMET.
To get something that runs fast, use -DCMAKE_BUILD_TYPE=Release or
-DCMAKE_BUILD_TYPE=RelWithDebInfo if you want to go fast and optionally if
you want debug info to go with it. Release mode makes a very large difference
in performance.
- Enabling GPU Support.
To enable support for Nvidia, AMD GPUs you need to set the respective option in
cmake:
# NVIDIA GPU
$ cmake ../ -DENABLE_NVIDIA_GPU_BACKEND=ON
$ make
# AMD GPU
$ cmake ../ -DENABLE_AMD_GPU_BACKEND=ON
$ makeThis will download and install Triton, a MLIR dialect for targeting GPUs used by COMET as a backend, and enable the GPU-related options in comet-opt.
You can also specify a default target device capability by passing the option
-DDEVICE_COMPUTE_CAPABILITY=<gpu-capability> in cmake. You can specify the same attribute later at the comet-opt command using the flag --gpu-compute-capability
Example options include sm_80, sm_90 for Nvidia and gfx908 for AMD. For example:
# Example for NVIDIA GPU
$ cmake ../ -DENABLE_NVIDIA_GPU_BACKEND=ON -DDEVICE_COMPUTE_CAPABILITY=sm_90
$ make
# Example for AMD GPU
$ cmake ../ -DENABLE_AMD_GPU_BACKEND=ON -DDEVICE_COMPUTE_CAPABILITY=gfx908
$ make- Enabling FPGA Support.
To enable support for FPGAs, (currently only Xilinx/AMD), you need to set the flag
-DENABLE_FPGA_TARGET=ONincmake. The FPGA support relies on other dependencies including an older version of LLVM found as a submodule intools/llvm-spirvand a LLVM-SPIRV translator found intools/spriv-llvm-translate. Setting the above flag will automatically download and install these dependencies, as well as MCL the runtime system used to issue interact with the FPGA. For more information see here.
This project is licensed under the Simplified BSD License. See the LICENSE file and the DISCLAIMER file for more details.
Issues with COMET can be reported through GitHub. We will try our best to timely address issues reported by users. The community is also welcome to discuss any remedies or experience that may help to resolve issues.
Contributions to COMET are welcome. The community can get involved by contributing some new feature, reporting bugs, and/or improving documentation. Please feel free to create a pull-request on GitHub for code contributions. We will try our best to timely incorporate user requests.
We encourage you to use GitHub’s tracking system to report any issues or for code contributions as mentioned above. For any other queries, please feel free to contact us via email:
- Gokcen Kestor (email: [email protected]), Pacific Northwest National Laboratory (PNNL), United States.
- Zhen Peng (email: [email protected]), Pacific Northwest National Laboratory, United States.
- Polykarpos Thomadakis (email: [email protected]), Pacific Northwest National Laboratory, United States.
- Ryan Friese (email: [email protected]), Pacific Northwest National Laboratory, United States.
If you use COMET in your research or work, please cite any of the following relevant papers:
- Erdal Mutlu, Ruiqin Tian, Bin Ren, Sriram Krishnamoorthy, Roberto Gioiosa, Jacques Pienaar & Gokcen Kestor, COMET: A Domain-Specific Compilation of High-Performance Computational Chemistry, In: Chapman, B., Moreira, J. (eds) Languages and Compilers for Parallel Computing, LCPC 2020, Lecture Notes in Computer Science, vol 13149, Springer, Cham. DOI and BIB.
@InProceedings{COMET:LCPC-20,
author={Mutlu, Erdal and Tian, Ruiqin and Ren, Bin and Krishnamoorthy, Sriram and Gioiosa, Roberto and Pienaar, Jacques and Kestor, Gokcen",
editor={Chapman, Barbara and Moreira, Jos{\'e}},
title={COMET: A Domain-Specific Compilation of High-Performance Computational Chemistry},
booktitle={Languages and Compilers for Parallel Computing},
year={2022},
publisher={Springer International Publishing},
address={Cham},
pages={87--103}
}
- Ruiqin Tian, Luanzheng Guo, Jiajia Li, Bin Ren, & Gokcen Kestor, A High Performance Sparse Tensor Algebra Compiler in MLIR, In: IEEE/ACM 7th Workshop on the LLVM Compiler Infrastructure in HPC, LLVM-HPC 2021, November 14, 2021, St. Louis, MO, United States. DOI
@InProceedings{COMET:LLVM-HPC-2021,
author={Tian, Ruiqin and Guo, Luanzheng and Li, Jiajia and Ren, Bin and Kestor, Gokcen},
booktitle={2021 IEEE/ACM 7th Workshop on the LLVM Compiler Infrastructure in HPC (LLVM-HPC)},
title={A High Performance Sparse Tensor Algebra Compiler in MLIR},
year={2021},
pages={27-38},
doi={10.1109/LLVMHPC54804.2021.00009}
}
The COMET compiler is supported in part by the Data-Model Convergence (DMC) initiative at the Pacific Northwest National Laboratory.
This work is also supported in part by the High Performance Data Analytics (HPDA) program at the Pacific Northwest National Laboratory.
This work is also supported in part by the U.S. Department of Energy’s (DOE) Office of Advanced Scientific Computing Research (ASCR) as part of the Center for Artificial Intelligence-focused Architectures and Algorithms (ARIAA).
