EMI & SPE Compiler Testing

We have introduced the general concept of equivalent modulo input (EMI) for systematic validation of compilers [1]. Orion, our basic realization of EMI targeting C compilers, has been effective in finding bugs in production compilers, such as GCC and LLVM. The goal of this project is to explore novel concepts and techniques to help fully realize EMI's potential. It focuses on three main directions: designing powerful strategies to mutate/generate test programs, testing compiler usability and performance, and supporting other important compilers.

We have maintained our continuous, extensive effort in stress-testing GCC and LLVM to benefit the entire community:

[GCC/LLVM bugs: 1,634 (total) / 1,076 (fixed)]
[Reports: GCC (link1, link2, link3, link4, link5), LLVM (link1, link2, link3, link4, link5)]

[Recent CompCert bug reports: 31 (total) / 27 (fixed)]
[Reports: link]

[Recent Scala and Dotty bug reports: 42 (total) / 17 (fixed)]
[Reports: link]

[Recent ICC bug reports: 35 (total) / unknown (fixed)]
[Reports: link]

Project Overview: [ slides, talk @ EPFL (video), talk @ ETH (video) ]

Publications

[1] Compiler Validation via Equivalence Modulo Inputs. [slides]
Vu Le, Mehrdad Afshari, and Zhendong Su.
In Proceedings of PLDI'14, Edinburgh, UK, June 9-11, 2014. (18%)

PLDI Distinguished Paper Award

[2] Randomized Stress-Testing of Link-Time Optimizers.
Vu Le, Chengnian Sun, and Zhendong Su.
In Proceedings of ISSTA 2015, Baltimore, MD, July 12-17, 2015. (27.7%)

[3] Finding Deep Compiler Bugs via Guided Stochastic Program Mutation.
Vu Le, Chengnian Sun, and Zhendong Su.
In Proceedings of SPLASH/OOPSLA, Pittsburgh, PA, October 2015.

[4] Finding and Analyzing Compiler Warning Defects.
Chengnian Sun, Vu Le, and Zhendong Su.
In Proceedings of ICSE 2016, Austin, TX, May 2016. (19%)

[5] Coverage-Directed Differential Testing of JVM Implementations. [talk video]
Yuting Chen, Ting Su, Chengnian Sun, Zhendong Su, and Jianjun Zhao.
In Proceedings of PLDI'16, Santa Barbara, CA, June 13-17, 2016. (16%)

[6] Toward Understanding Compiler Bugs in GCC and LLVM. [Artifact Download]
Chengnian Sun, Vu Le, Qirun Zhang, and Zhendong Su.
In Proceedings of ISSTA 2016, Saarbrucken, Germany, July 18-20, 2016. (25%)

[7] Finding Compiler Bugs via Live Code Mutation. [talk video]
Chengnian Sun, Vu Le, and Zhendong Su.
In Proceedings of SPLASH/OOPSLA, Amsterdam, October/November 2016. (25.6%)

[8] Skeletal Program Enumeration for Rigorous Compiler Testing.
Qirun Zhang, Chengnian Sun, and Zhendong Su.
In Proceedings of PLDI, Barcelona, Spain, June 2017. (14.6%)
arXiv version: CoRR, ACS/1610.03148, 2016.

[9] Perses: Syntax-Guided Program Reduction.
Chengnian Sun, Yuanbo Li, Qirun Zhang, Tianxiao Gu and Zhendong Su.
In Proceedings of ICSE 2018, Gothenburg, Sweden, May 2018. (21%)

Acknowledgments

This project was supported in part by NSF CCF Grant 1528133 and a Google Faculty Research Award. We gratefully acknowledge the support.

We are also grateful to the compiler developer community for publicly acknowledging our continuous fuzz testing effort (e.g., in the LLVM release notes and in GCC's list of contributors).