Measurement process
The measurement of software energy consumption requires a systematic and repeatable measurement process aimed at producing comparable, reliable, and practically applicable information on software energy usage in various operating environments. The measurement cases in this benchmark databases are methodologically based on this measurement process, which is described at a general level in the Visiiri project’s self-study materials and in greater detail in the measurement laboratory’s documentation. The primary goal of the process was to create a unified framework that allows the energy efficiency of software to be assessed consistently, regardless of the specific software, hardware, or usage scenario under consideration.
The development of the measurement process was iterative and research-driven. The requirements specification for the process were based on expert interviews as well as previously conducted benchmarking and measurement examples. These have helped identify key challenges in practical measurement work, such as measurement repeatability, environmental heterogeneity, and the management of background factors affecting energy consumption. Combining expert knowledge with empirical measurement examples enabled the development of a process that meets the needs of both research and practical software development.
In the design of the measurement process, an assessment of the hardware, software, and the measurement methods themselves was conducted in parallel. This was necessary because the software’s energy consumption depends heavily on both the hardware used and the structure of the software stack. The process was refined iteratively so that the measurement method would be as universally applicable as possible and suitable for various use cases. The goal was not merely to conduct individual measurements, but rather to build a measurement infrastructure that supports large-scale and continuous assessment of energy efficiency.
From the perspective of generalizability, the process was designed to comprehensively cover a wide range of client and server devices, as well as their operating environments. This enables, for example, the comparison of different software versions, database solutions, programming languages, or device architectures with one another. At the same time, the process took into account the need to compare comparable software and hardware types in such a way that the measurement results remain technically sound and analyzable. Comparability requires control of measurement environments, consistent load scenarios, and precise management of log and metadata.
A key aspect of the process was a high degree of automation. Performing measurements must be as effortless and repeatable as possible, even in varying environments. For this reason, efforts have been made to define the measurement process through clearly documented steps so that measurements can be performed consistently regardless of the operator or the operating environment. Automation typically covers the initiation of measurements, load management, energy data collection, log data storage, and report generation. In addition, the data generated during the process can be automatically exported to reporting systems and log databases for later analysis. This supports both research use and long-term monitoring during the various stages of software development.
In addition, the process was designed to be scalable to various types of equipment and integrated into modern software development practices, such as DevOps and CI/CD pipelines. This enables automatic monitoring of energy efficiency, for example, between software versions or as part of tests performed during continuous integration. In the long term, this approach supports the systematic development of more energy-efficient software and the assessment of the sustainability of software production as part of a broader framework for responsible software development.
Measurement scheme implementation methods
The following is a brief overview of the key benchmarking methods in the benchmark database that are suitable for various types of applications (frontend, backend, full-stack, mobile, etc.).
Script driven measurement
The simplest type of software to measure is a command-line program. In such cases, the measurement can be carried out using a straightforward script that coordinates the measurement session and runs, and executes the program being measured. A separate measurement case is defined for each function to be measured. The script communicates with the measurement software before and after the execution, allowing the timestamped measurement data to be linked to the correct measurement run. If the script and/or the program being measured require a complex runtime environment, it may be advisable to isolate the measurement using container technologies. Similarly, if the program being measured is already isolated in a container, it may also be necessary to bring the measurement script into the container to simplify the setup.
Container based measurement
Container technologies enable applications to run in an isolated environment. Containers contain the software components and environment configurations required by the software, separate from other containers and the host system, which facilitates the use of individual services and service suites in particular. The same measurement can be performed in multiple environments using the same software configuration without differences in the operating system or installations significantly affecting the results. In addition to improving measurement repeatability, this also facilitates the sharing of measurement setups across organizations. The goal is generally to keep container startup and shutdown outside of measurement runs, as these can be relatively resource-intensive operations and are often not part of the software’s normal usage.
Measurement of backend services
There are several ways to monitor backend services. At its simplest, the service is treated as a “black box,” in which case an external script sends requests to the service without analyzing its internal operations. This may be the only possible approach, for example, in closed-source or cloud-based systems. Another method is to integrate measurement coordination into the service’s program code. In this case, the start and end markers for the measurement can be placed, for example, directly around the implementation of the web service’s API calls. This allows the measurement to be more precisely targeted at the execution of a specific function. The service to be measured is usually started before the measurements, after which individual requests or usage scenarios are executed during the measurement runs. If desired, it is also possible to measure the service startup or the processing of the first request.
Measuring user interfaces with software robotics
Software robotics can be used to test graphical user interfaces, allowing the interface’s functions to be performed automatically and reproducibly in the same way as when controlled by a human user. One of the best-known software robotics tools is Selenium, which can be used to build repeatable testing scenarios for web applications to estimate energy consumption. A Selenium session and the browser it controls are typically either kept running for the entire measurement session or moved outside the scope of individual measurement runs, since initializing the session is a relatively resource-intensive operation compared to individual page loads and, in particular, individual actions on a page that has already been loaded. Measuring mobile devices often requires special arrangements. Application automation can be based, for example, on user interface automation, programmatic interfaces, or control mechanisms added to the application. In some cases, measurement coordination can be implemented, for example, using client-side code that communicates with the measurement software over a network connection.
Measurement reports
The measurement reports form the core user interface of the measurement database, which is designed to provide both a general overview and a technically detailed view of the measurement results for software energy consumption. The reports are designed so that the same measurement data serves several different user groups: software developers, researchers, educational institutions, and organizations that evaluate the energy efficiency of their systems.
The purpose of the reporting view is not merely to present individual measurement results, but also to support the interpretation, comparison, and further analysis of measurement data. In particular, when it comes to energy measurements, a single value is rarely sufficient without context, as energy consumption depends heavily on the measurement environment, the load, the hardware, the timing, and the underlying software and operating system components.
User interface
At the top of the report interface are three main options: Language, Case, and View. The language setting changes the language of the report interface and interactive components.
The Case option determines which measurement case is being viewed. By default, there is only one case for PowerGoblin's reports, but reports from different measurements can be compiled into a shared database from which to select the desired case. For example, using our benchmark database, the user can select a measurement related to databases, web services, container technologies, or mobile computing. All graphs, tables, and analyses are updated according to the selected case.
The View selection offers two different levels of reporting views: High-Level Report and Technical Report. The former focuses on the measurement’s area of application, the key results, and summarized conclusions. The goal is to provide an easily understandable summary even for users who do not have in-depth background in energy measurements or software system performance analysis.
The Technical Report includes a more detailed description of the measurement system and the measurement methodology. The report may also describe factors related to measurement repeatability, such as buffer handling, temperature stabilization, control of background processes, and calibration of measurement equipment. These factors are particularly important in energy measurements, where even small changes in the environment can affect the results. The report may include, for example: a description of the physical and logical architecture, the hardware and software versions used, operating system settings, measurement device configurations, a description of the source data, the structure of automation scripts, measurement parameterization, literature references, and known limitations and sources of error.
Boxplots
The box plot serves as the default view in the reporting tool because it offers a concise and statistically informative way to compare different measurement targets. In energy measurements, a single measurement result is usually not sufficient to describe the system’s behavior reliably; instead, multiple repetitions are needed to assess variability.
A boxplot allows for a quick visual analysis of data from a statistical perspective. It makes it possible to identify data dispersion, outliers, and to compare the "superiority" of different technologies. The graph makes it easy to see, for example, which solution consumes the least or clearly the most energy on average, which solutions show high variability, or how stable the measurement target is across different measurement runs
Available metrics include, for example: voltage, current, power, total energy, measurement duration, CPU load, memory usage, network traffic, and disk operations. In more complex measurement scenarios, multiple metrics can be combined into a single view. For example, in database measurements, you can simultaneously view energy consumption, disk operations, and memory usage, which provides a clearer picture of what drives the energy consumption.
A key advantage of graphing in energy measurement is the ability to highlight the natural variation in measurements. In practice, measurement data is almost never completely deterministic, as operating system timing, caches, background processes, temperature, and the hardware’s internal optimization mechanisms cause variation.
Tilastollisen analyysin näkökulmasta on tärkeää huomioida:
- mittauskertojen riittävä määrä
- poikkeavien havaintojen tunnistaminen
- normaalijakaumaoletuksen mahdollinen puuttuminen
- mediaanin käyttö keskiarvon sijaan epäsymmetrisissä jakaumissa
- hajonnan merkitys käytännön toistettavuudelle
Mikäli vaihtelu on suurta, voi se viitata:
- epävakaaseen mittausympäristöön
- käyttöjärjestelmän taustaprosesseihin
- lämpötilan vaikutuksiin
- välimuistien hyödyntämiseen
- ajoittaiseen verkkokuormaan
- satunnaisiin virhetilanteisiin
Raportointityökalu mahdollistaa myös muiden casejen datan tuomisen vertailuun. Tämä mahdollistaa esimerkiksi:
- eri teknologia-alueiden vertailun
- eri laitteistojen vertailun
- ohjelmistoversioiden väliset analyysit
- optimointien vaikutusten tarkastelun
Summary table
A table of results provides the same data as a boxplot, but in numerical form. The table includes statistical minimums, maximums, means, and standard deviations, as well as the name of the measured variable and its description. When evaluating energy consumption, the mean alone is often not a sufficient indicator. Especially in software measurements, distributions can be skewed or contain random spikes, in which case the median and measures of dispersion are often more informative. The table can be used to identify potential errors, such as missing measurement runs or meter readings, unrealistic values, and individual outlier measurements.
The tabular format allows for a somewhat more detailed analysis than graphs, and the aggregated results data can be used as-is in reporting. If you wish to analyze the results data in greater detail using external statistical tools, we recommend reading the metric data from the JSON-formatted raw data stored in the database.
The reporting tool does not currently aim to assess environmental impacts directly from measurement data. For further analysis, the measurement data must be integrated into a separate environmental impact model if the goal is to assess, for example, the carbon footprint or emission impacts. Energy consumption alone does not reveal the software’s environmental impacts, as the overall impact depends on factors such as the method of energy production, the operating environment, the hardware’s lifecycle, the number of users, the utilization rate, the energy efficiency of the data center, and the cooling systems.
Scatter plots
A scatter plot allows you to examine individual measurement runs over time. Unlike a box-and-whisker plot, which summarizes results in a statistical format, a scatter plot shows the dynamic behavior of the measurement and instantaneous changes.
By default, the scatter plot displays power as a function of time, because instantaneous power consumption often reveals more about the software’s internal behavior than just total energy. Furthermore, many performance-related issues manifest themselves specifically as momentary power spikes or uneven resource utilization. The graph is therefore well-suited for detailed analysis of measurement runs.
Pistekuvaajassa voidaan tarkastella esimerkiksi:
- tehoa
- prosessori(e)n käyttöastetta (0-100%), jossa 100% tarkoittaa kaikkien prosessorien / ytimien olevan käytössä
- muistinkäyttöä
- verkkoliikennettä
- levyoperaatioita
- prosessikohtaisia metriikoita
Esitystapa riippuu tarkasteltavasta suureesta. Esimerkiksi:
- vapaan muistin määrä esitetään absoluuttisena arvona
- verkkoliikenne esitetään käyttöjärjestelmän laskuriarvojen erotuksena
- levyoperaatiot näytetään siirtonopeutena
- prosessorikuorma prosenttiosuutena tai aktiivisten prosessien lukumääränä (load avg)
Riittävän hienojakoinen mittaridata mahdollistaa jopa ohjelmiston sisäisen toiminnan analysoinnin. Tietyt algoritmit, garbage collection -syklit, tietokantakyselyt tai renderöintivaiheet voivat näkyä selkeinä tehopiikkeinä mittausdatassa.
Pistekuvaaja on erityisen hyödyllinen poikkeavien mittausten tunnistamisessa. Mikäli eri mittausajot eroavat toisistaan merkittävästi, voi syynä olla esimerkiksi:
- välimuistin hyödyntäminen
- käyttöjärjestelmän optimoinnit
- verkkoyhteyden vaihtelu
- taustaprosessit
- epäonnistuneet mittausajot
- lämpötilan nousu
- suorittimen throttling
- levyvälimuistit
- rinnakkaisten prosessien vaikutukset
Mittauksen toistettavuuden kannalta on tärkeää hallita muuttuvia tekijöitä ennen mittausten suorittamista. Tämä voi tarkoittaa esimerkiksi:
- levyvälimuistien tyhjentämistä
- prosessorin taajuuden vakiointia
- lämpötilan tasaamista ennen ajoja
- taustaprosessien minimointia
- verkkoyhteyksien vakiointia
- automaattisten päivitysten estämistä
Pistekuvaaja mahdollistaa myös mittausten vaiheistamisen analyysin. Esimerkiksi ohjelmiston käynnistys, datan lataus, laskenta ja tulosten tallennus voivat näkyä erillisinä vaiheina, joiden energiankulutusta voidaan tarkastella toisistaan riippumatta.
Tilastollisen analyysin näkökulmasta pistekuvaaja auttaa arvioimaan:
- mittausdatan stationaarisuutta
- ajallista korrelaatiota
- lämpenemisen vaikutuksia
- transienttien merkitystä
- mittauskohinan määrää
- mahdollisia systemaattisia virheitä
Erityisesti energiamittauksissa ajan suhteen tarkasteltava data on usein tärkeämpää kuin yksittäinen kokonaislukema, koska ohjelmistojen energiankulutus muodostuu useiden erilaisten toimintavaiheiden yhteisvaikutuksesta.