Talk slides:
Paper link:
Paper PDF @QRS’16 conference: click here
Approach Overview:
NOTICE (as NOn-functional TestIng of CompilErs) is a component-based framework for non-functional testing of compilers through the monitoring of generated code in a controlled sand-boxing environment.
NOTICE is an on-demand tool that employs mono and multi-objective evolutionary search algorithms to construct optimization sequences that satisfy user key objectives (execution time, code size, compilation time, CPU or memory usage, etc.). It is based on micro-services to automate the deployment and monitoring of different variants of optimized code.
In general, there are many non-functional properties that can be influenced by compiler optimizations, e. g., performance (execution time), code quality, robustness, etc. In this work, we focus on the efficiency of optimized code in terms of resource consumption (memory and CPU). To assess the performance of optimized code many configurations, i.e., optimizations, must be applied to software programs. Running different applications with different configurations on one single machine is complicated because a single system has limited resources and it can lead to performance regressions. Therefore, we need to deploy the test harness, i. e., the software artifacts needed to perform a test, on an elastic infrastructure that provides to compiler user facilities to ensure the deployment and monitoring of different configurations of optimized code. For this purpose, we propose NOTICE, a non-functional testing infrastructure based on System Container techniques such as the Docker1 environment. This framework automates the deployment and execution of applications inside software containers by allowing multiple program configurations to run autonomously on different servers (i. e., a cloud servers). It also provides a distributed environment where system resources can be finely managed and limited according to the needs. We rely on this component-based infrastructure and benefit from all its advantages to:
- Deploy generated code within containers.
- Automate optimization sequence generation.
- Execute and monitor service containers.
- Gather performance metrics (CPU, Memory, I/O, etc.).
To summarize, we present, in Figure 3, an overall overview of the different components involved within NOTICE. Our testing infrastructure will run different jobs within Docker containers. First, in the top level layer, we use NOTICE to generate different configurations of code using compiler optimizations (e.g., GCC compiler). Then, we wrap generated code within multiple instances of our preconfigured Docker image. Each container will execute a specific job. For our case, a job represents a program compiled with a new optimization
In this work, we aim at providing a new alternative for choosing effective compiler optimization options compared to the state of the art approaches. In fact, since the search space of possible combinations is too large, we aim at using a new search-based technique called Novelty Search [11] to tackle this issue. The idea of this technique is to explore the search space of possible compiler flag options by considering sequence diversity as a single objective. Instead of having a fitness-based selection that maximizes one of the nonfunctional objectives, we select optimization sequences based on a novelty score showing how different they are compared to all other combinations evaluated so far. We claim that the search towards effective optimization sequences is not straightforward since the interactions between optimizations is too complex and difficult to define.
Generally, NS acts like GAs. However, NS needs extra changes. First, a new novelty metric is required to replace the fitness function. Then, an archive must be added to the algorithm, which is a kind of a database that remembers individuals that were highly novel when they were discovered in past generations.
NOTICE - A Framework for Non-functional Testing of Compilers 