Benchmark database
The purpose of the benchmark database developed in the Visiiri project was to identify, demonstrate, and evaluate various methods for measuring the energy consumption of ICT systems through practical case studies. The starting point for the work was the definition of requirements for various measurement cases (cases) so that they comprehensively cover various software, hardware, and architectural solutions, as well as the appropriate measurement methods. Detailed results data, documentation, and analysis have been produced for each case, and these have been used to compile a unified database to support the measurement and comparison of energy consumption.
The overarching goal of the benchmark database is to identify and develop methods for measuring the environmental impact of the ICT sector, with the aim of improving the energy efficiency of ICT systems. The focus is on the systematic measurement of power consumption in ICT system hardware and the modeling of the impacts of software components. For this reason, efforts have been made to design the measurements to be as objective, reproducible, and systematic as possible, so that the results are suitable both for scientific research and for supporting technical and business decision-making in software development.
The documentation also examines how measurement activities can be integrated into software development processes. The goal is to enable continuous monitoring and optimization of energy consumption during system development and maintenance. With regard to the methods, their practical applicability has been assessed as part of the software development process, continuous integration, and quality assurance of performance and energy efficiency.
In compiling the database, we also examined the comparability of the energy efficiency of software and technologies commonly used in the ICT sector. Because the number of possible software and technology combinations is extremely large and constantly evolving, it is simply not possible to cover all options. For this reason, the guiding principle in designing the case studies was generalizability, so that they can also be used to evaluate and compare other similar software components.
The design of the case studies emphasizes the experimental exploration of measurement techniques. The goal is to identify various measurable targets as well as the appropriate measurement methods and practices for them. Systematic measurement of energy consumption from a software development perspective remains a relatively underutilized methodology, which is why measurement practices, measurement environments, and the tools and equipment to be used had to be defined and built as part of the project’s research work.
The documentation also takes into account the impact of different types of hardware and software on measurement and energy efficiency. The study examines both physical measuring devices and software-based measurement solutions, as well as their suitability for various applications. At the same time, the generalizability, reliability, and practical applicability of the measurements are assessed for future research, development, and decision-making contexts.
The measurements in the benchmark database are divided into four main categories:
- Frontend measurements: These measurements focus on the end user device and the energy consumption of its software components, for example through software robotics or API calls.
- Backend measurements: measurements are performed on the backend server or in server-side software, typically in environments based on container and interface technologies.
- Full-stack measurements: combines the aforementioned front-end and back-end components and also examines the energy efficiency of data transfer between different parts of the system measurements targeting mobile devices, which examine the energy consumption of mobile technologies and their relationship to, for example, cloud-based implementations
This section also includes an analysis and conclusions regarding the measurements conducted, the methodologies, equipment, and software used, as well as the significance of the measurements for future research and development. In addition, based on the measurement results, modeled estimates of the systems’ sustainability and environmental impacts, such as the carbon dioxide impacts of energy consumption, so that the measurement data can also be utilized to support decision-making regarding sustainable software development and responsible ICT operations.