The Sustainability Assessment Framework Toolkit: A Decade of Modeling Experience (2024)

1 Introduction

Software intensive systems play a crucial role in most, if not all, aspects of modern society. As such, both their own sustainability and their role in supporting sustainable processes, must be realized by design.

From an environmental perspective, the energy footprint of digital solutions is worrisome. For example, the Energy Efficiency Directive for Data Centers[1], reports that in the EU, from 2010 to 2018 data centre energy consumption increased by 42% and is forecast to further increase by 28.2% by 2030, representing about 3.2% of the EU final electricity demand. Reducing the energy demand of ICT, including data centres, is an important step in the target reduction of overall GHG emissions of 55% by 2030 compared to 1990 levels. Given that nowadays most software is to various degrees cloud based (e.g.,[2, 3]), its energy efficiency is imperative. If we also consider the global adoption of AI-based solutions like generative AI[4], there is growing evidence that the energy footprint of software will escalate in the coming years[5].

Next to the environmental perspective, sustainability entails other dimensions (see Section2) which further increase the role of software in society at large: from a social perspective software usability, for example, can support or hinder accessibility to fundamental services like healthcare and education; from an economic perspective, affordability of software products and related technologies can significantly influence economic growth; and of course from a technical perspective, evolvability or integrability of software to accommodate over time the changes in society and consumer needs, can act as the motor for, or against, innovation.

As sustainability and its dimensions have a broad scope and are interdependent, software professionals and decision makers need to consider software sustainability at a high level of abstraction. This way, the role of software elements and the way they influence one another can be (re-)designed for, monitored and reasoned about. The software architecture level provides the right abstraction needed for the above.

In this paper, we present the Sustainability Assessment Framework (SAF) Toolkit – a set of instruments that aim to support software architects and design decision makers in modeling sustainability as a software quality property. The SAF Toolkit is the result of our experience gained in over a decade of cases in collaboration with industrial partners.

The intended target audience of this paper are (i) practitioners responsible both for setting priorities bringing software sustainability to the forefront of their organizations and for deciding on how sustainability concerns are designed in software solutions, and (ii) instructors willing to include software sustainability into Computer Science and Software Engineering programs. Both can access the SAF Toolkit guidelines, tutorials, and links to open-source instruments in the supplementary online material[6].

We illustrate the toolkit with examples stemming from various cases. We extract our lessons learned, and our current research and future plans to extend the SAF Toolkit for further architecture modeling and measurement.

The remainder of this paper is organized as follows.Section2 presents the background on the notion of sustainability, its relation to software as used in our work, and the notion of key performance indicator, while Section3 presents related works.Section4 describes the SAF Toolkit and its instruments in detail, and explains the current release and its evolution over time, and the next steps for future research.Finally, in Section5 we discuss additional overarching observations, and conclude the paper.

2 Background

In this section, we provide the background knowledge necessary for the remainder of this paper. We introduce the notion of sustainability in general and in the context of software. We provide an overview of sustainability in terms of the two dimensions that we find indivisible in a sound definition of sustainability, namely the dimension expressing the various sustainability aspects that are within the focus of a software intensive system, and the dimension expressing the related impact over time. We then close with the concept of Key Performance Indicator.

3 Related Work

In this section, we discuss the related works that focus on modeling frameworks or assessment initiatives aimed at achieving software sustainability.

Naumann etal. [26] present a reference model for Green and Sustainable Software – the GREENSOFT model. This model is designed to capture sustainability effects (first-, second-, and third-order of effects) of software from their inception until their end-of-life. The model is supplemented by metrics for sustainability criteria (common, direct and indirect). The model also provides procedure models for the development, purchase, operation and use of software in a sustainably-sound manner. It rounds off by providing recommendations and tools for software development, purchase, administration and usage. The GREENSOFT model focuses on both Green IT and Green by IT.

Cabot etal. [27] use the i* framework for integration of sustainability design concerns in (i) design alternative selection for software, and (ii) decisions for business activities. The use of this framework is demonstrated through a case study by identifying sustainability-specific soft goals, tasks, agents and roles. Dependencies and relationships are identified between these elements. Goals are linked to each other with the relation of positive (‘help’) or negative (‘hurt’) impacts. The study acknowledges the challenges of the proposed approach as (i) difficulty in determining the appropriate level of detail for effectively representing sustainability, (ii) the need for a qualitative approach incorporating measurable measures to enhance the usefulness of the analysis, (iii) loose semantics of the proposed model, leading to consistency issues when different people use it, and (iv) lack of validation mechanism with stakeholder involvement for validation of decisions and their impacts.

Penzenstadler and Femmer [28] present a meta-model that is instantiated as a sustainability model to analyze sustainability dimensions, value indicators, and support activities. The meta-model describes the sustainability dimension as an aspect of the goal. Indicators are used to quantify a value that represents certain sustainability dimensions. These indicators, guided by external regulations, guide the activities that lead to achieving sustainability. It is, however, challenging to identify possible conflicts between goals. To solve this issue, Saputri and Lee [29] present an extension of this meta-model by including quality attributes as a measurement mechanism for indicators. A template is provided to carry out this trade-off analysis between goals by capturing the influence (conflict, support, neutral).

Duboc etal. [30] present a framework to raise awareness on the sustainability impacts of software systems in the context of requirements engineering. They present a question-based Sustainability Awareness Framework (SusAF) supplemented by Sustainability Awareness Diagrams. This framework aims to facilitate discussions, for it to become a de-facto standard in the industry as a thinking tool for short, medium, and long-term sustainability impacts of socio-technical systems [31].

More recently, the GAISSA project[32] provides architecture‐centric methods for the modeling and development of green AI‐based systems. It focuses on developing a greenability quality model for assessing AI models and systems, providing green-aware architecture-centric methods to data scientists and software engineers, and supporting analysis and decision-making for greenability improvement.

Guldner etal. [33] provide a reference measurement model called the Green Software Measurement Model (GSMM). The model uses existing research to provide recommendations, processes, and tools to help evaluate and minimize software’s environmental impact. The study presents a catalog of metrics to evaluate software’s energy and resource consumption. Further, it provides a list of tools for automation, resource logging, analysis, and fully integrated solutions. Hence, GSMM aims to aid practitioners in developing and evaluating software products for resource and energy efficiency.

The frameworks and models found in the literature facilitate capturing and representing software sustainability in different stages of the Software Development Life Cycle. With the SAF Toolkit, we envision support along certain aspects, specifically in the context of software architecture by (i) measuring the Quality Requirements representing design concerns via metrics (already supported by the current release of the SAF Toolkit), (ii) establishing a link of design concerns to software architecture and its elements (partially supported by the SAF Toolkit), (iii) connecting the low-level metrics measuring the Quality Requirements with high-level goals represented by Key Performance Indicators (KPIs) (in the future release of SAF Toolkit).

4 The SAF Toolkit

To provide a thorough introduction to the SAF Toolkit, we organize this section in three parts. First, we take a bird’s eye view of the toolkit to understand all instruments involved and how they interact. Then, we zoom into each instrument individually to examine their elements and show how the instrument has evolved over time to the current release of the toolkit. Finally, we outline our plan for the future and how we envision the planned extensions.

4.1 The Big Picture

Figure1 provides an overview of all the instruments that are part of the SAF Toolkit and those that are planned as future extensions. We refer to this view as the SAF Toolkit Overview, as it presents the instruments at a high level of abstraction and illustrates their inter-relationships.

In its current release (see inside the dashed blue rectangle), the SAF Toolkit consists of its two main instruments, namely the Sustainability-Quality (SQ) Model and the Decision Map (DM), both adopting a shared definition of the sustainability dimensions. This definition is used to classify the quality requirements from the perspective of the sustainability dimension to which the quality requirements relate.

In particular, to frame and illustrate sustainability-relevant design and quality requirements (QRs), the visual notation of Decision Maps (DMs) is utilized. DMs serve as a thinking framework to capture, reason, and uncover dependencies among sustainability-related QRs in the form of a visual diagram. Such visualization is also intended to foster communication among software architects regarding the sustainability of their software-intensive systems.

To consolidate definitions about the identified concerns, the Sustainability-Quality (SQ) Model encompasses a collection of Quality Attributes (QAs)¹¹1In the context of the SAF Toolkit, we use interchangeably terms quality requirement and quality attribute depending on the context of the discussion. This follows the observation by deBoer and van Vliet [35], that quality requirements become quality attributes during the design phase of a given software project. grouped into the four sustainability dimensions. QAs are derived in accordance with the ISO/IEC 25010 standard [36] and are based on QAs identified in prior projects [37, 38]. The SQ Model operationalizes the identified QAs provided by the DM by assigning a set of metrics and measures.

Research indicates that sustainability necessitates a careful assessment of trade-offs but within distinct Sustainability Dimensions [17, 39, 37]. As mentioned in Section2.2, different disciplines feature varying types and numbers of dimensions. The SAF Toolkit, with its DM and SQ Model aligns with the four sustainability dimensions as proposed by Lago etal. [17]. Nonetheless, while all four dimensions should be taken into account during the design phase to achieve balance, it is not always necessary for every dimension to be relevant for a specific software system [37].

Section4.2 delves deeper into the current release of the SAF Toolkit by further explaining each instrument.

The instruments yet to be integrated (see outside the dashed-blue rectangle in Figure1) include the KPI Model and the Software Architecture standard. To establish and define the metrics and measures for the QAs within the SQ Model in a systematic way, we aim to incorporate a KPI Model. Such a model will assist us in identifying sound KPIs relevant to the specific software project at hand.KPIs can emerge from two distinct approaches, top-down or bottom-up.In a top-down assessment, we envisage that KPIs can be formulated by starting with a sustainability goal derived from the DM, followed by gathering the relevant metrics and measures.Conversely, in a bottom-up assessment, concrete metrics from the SQ Model would be used to formulate, or associate to, a particular goal.Overall, once comprehensive KPIs are established in a systematic and traceable manner, they can provide essential feedback on QAs and related design decisions through tangible measures.

To contextualize the software architecture of a given software-intensive system, we aim to enhance the SAF Toolkit by positioning it with respect to the Software Architecture ISO/IEC/IEEE 42010 Standard [40] and explicitly integrating the relevant architecture elements. If present, this integration would enable us to link the QAs from the DM with specific architecture elements, such as software services or components. This connection is crucial for (i) understanding where to measure the previously defined KPIs and consequently (ii) determining the impact of a specific architectural element on a particular QA.
The detailed conceptualization and integration for the planned extensions are elaborated upon in Section4.3.

4.2 The SAF Toolkit: Where we are now

The current release of the SAF Toolkit produces two main outputs, the Decision Map and the SQ Model. To do so, the toolkit user is provided with a set of so-called instruments, i.e., tools providing the necessary guidance.

The following explains both outputs and for each of them, describes (i) the conceptual elements they frame, (ii) the visual elements with examples from some of our past projects, and (iii) the instruments guiding their creation. We close with a reflection on the experience and the points for improvement related to each instrument.

4.2.1 Decision Map and its Instruments

The Conceptual Elements

In Figure2, the key elements of the DM (coloured in green) are encompassing architecture-related concepts widely adopted in the field, as well as the essential sustainability-related concepts.

The architecture-related concepts include class Software Architecture as the generic anchor to the software project under study (or if you wish, the generic placeholder representing the software architecture), and class Design Concern representing the set of architecture design concerns that are being addressed. In addition, a DM may include elements of class Feature (see the bottom and left-hand side of Figure2), which according to the ISO/IEC 25010 standard is an aspect or property (functional or non-functional) realized by architecture elements and as such part of the software architecture.In the current release of the SAF Toolkit, a DM can be designed to illustrate design concerns that may address both Sustainability-related Requirements (e.g., in EV charging services, how to ensure that any type of citizen can easily understand how to charge), and Quality Attributes (e.g., usability).

The sustainability-related concepts, in turn, are represented by association class Effect Type and class Expected Impact. The Effect Type represents the type of sustainability effect between a pair of design concerns (e.g., the effect between performance and energy efficiency). Such effect can be measured (or designed to be) positive (i.e., contributing to sustainability), negative (i.e., hindering sustainability) or undecided (i.e., if no measures are taken yet and at design-time it is unclear what type of effect will be observed).The Expected Impact, in turn, represents the sustainability time dimension and is meant to frame if the impact of a certain design concern is immediate (first order), enabling (second order) or systemic (third order).

The Decision Map

By focusing on a selected software project, the DM can illustrate the features that should be sustainability aware. As shown in Figure3, a DM makes use of a relatively simple visual notation (summarized in Figure3.(a)) so that any type of stakeholder, from IT specialists to general decision makers, can use it to reason about (i) the sustainability implications of their design decisions, and about (ii) the decisions’ effect on QAs in the short- and longer term.

As an example, Figure3.(b) shows the DM for a smart lighting software architecture project, and in particular the feature customize lightning and the related network of sustainability effects that such feature is expected to have on various relevant QAs. For example, the feature is expected to have a positive effect on energy savings in the environmental Sustainability Dimension (i.e., one can avoid electricity waste by tuning lightning to the actual user needs), which in turn would have a positive effect on energy costs in the economic Sustainability Dimension (i.e., by saving energy one would reduce energy costs, too).

Further, QAs are mapped on the Time Dimension to enable explicit representation of the type of sustainability impacts that they are designed for. In our example in Figure3.(b), configurability of lightning to human behavior has proven to enable positive (second order) effects on well being (e.g., on sleeping and eating patterns of children) which in turn may have systemic (third order) effects on the need for (and cost of) healthcare.

The guiding Instruments

The SAF Toolkit includes a number of instruments that provide guidance for creating a DM. These instruments are the Checklist and the Decision Graph, both described in the following.

The Checklist provides a list of reflective questions that help guiding the creation of a DM in a systematic way. It mainly focuses on the identification of the most important QAs and design concerns. Figure4.(a) shows a fragment of the Checklist with the reflective questions for identifying the main QAs (and the types of related sustainability effects) for the software project being considered.

The Decision Graph provides guidance to classify each QA according to the most relevant Time Dimension. Practically,it helps deciding where each selected Design Concern (DC) should be placed among the three impact levels (i.e., immediate or first-order, enabling or second-order, systemic or third-order – the graph leaves in Figure4.(b)).
In particular, the black text in the figure exemplifies the possible interpretations of each impact level; the blue text provides a simple example of possible mappings. As such, if the QA would be maintainability, depending on its definition, its impact could be measured directly in the software system (e.g., if defined as the level of modularity of the design), indirectly (e.g., if defined as reusability and hence measurable only after the system is reused in new projects), or systemic (e.g., if defined as durability and hence measurable in terms of the stability of the deployed system over longer time periods).

Reflections

It must be noted, that the DM notation has evolved over the years to address the needs of sustainability-aware software architecture design decision making, thanks to the experience gained via its use in a series of diverse cases provided by external partners, mostly from industry. For example, while the toolkit was created starting in 2015, only in 2020 we introduced the notion of ‘feature’ to make explicit which aspect of a software system would be responsible for which QAs. Accordingly, before 2020 the DM in Figure3.(b) was not including any feature and was looking like the one in Figure5.(a)²²2Notice that before 2020 also the visual notation for the effect types was different..

As another example, the more recent work reported in Funke etal. [41] had to represent the fact that some features can be realized by multiple variants as illustrated in Figure5.(b).

In Table1 we summarize a list of exemplary cases that we carried out over the years to apply the SAF Toolkit and when needed, improve it according to the lessons we learned. On the right-hand side, we point to when the SAF Toolkit was revised to accommodate needs that we deserved as generally applicable and adding value. In particular, changes to the DM have been:

• •

  In 2017, a case study carried out with the KPMG consultancy company revealed that after carrying out empirical experiments, we could decorate the effect-arrows with labels that would illustrate the measured impact e.g., of a quality concern like performance on another quality concern like energy efficiency. As a consequence, we added impact labels to the DM visual notation (see comment in Figure3.(a)).

• •

  In 2020, we carried out a research project on food security, in particular in resource-constrained environments (like some African countries) were access to software services can help sustainability and equity. We had to model which features were meant to address equity requirements (e.g., provide farmers with audio-based access to services in various local dialects) so that all farmers could exploit the opportunities of selling online their products. It must be noted that in the same project, we experimented with modeling with goals in DMs, too, i.e., making explicit the sustainability goals that features are meant to realize. While in this specific project this modeling exercise was beneficial, we did not encounter such a modeling need afterwards, hence we did not leverage this in the toolkit yet.

• •

  In some other projects between 2020 and 2022, we found it useful to, e.g., model design concerns that are in a hierarchical relation (like that existing between quality characteristics and sub-characteristics) or in the form of thematic clusters (like complexity into organisational, technical or social complexity). While such representations could be model kinds of DMs, we consider them as different views. Also in these cases, we did not need to illustrate them in any other project and hence refrained to extend the DM notation.

• •

  In 2020-2023 we carried out a yearly winter school attended by both researchers and practitioners. The participants would bring their own case study. Every school edition, we had one case where modeling the DM elements pertaining a stakeholder type would be beneficial (e.g., if some quality concerns would pertain producers and some others consumers). While also in this case we refrained from leveraging the clustering of concerns in the standard DM notation, we noted down this experience as a lesson potentially useful for future releases of the toolkit.

YEARTOPICSHORT DESCRIPTIONCASE ORIGINREFS.SAF TOOLKITIndustryAcademiaDMSQ Model2016SoSA exerciseTutorial at International Conference on Advanced Information Systems Engineering (CAiSE) 2016, where three working groups used a smart mobility case to critically evaluate DMs.✓[42]✓2017KPMG QubusEmpirical experiment assessing the impact on energy consumption and performance of different releases and deployment strategies of an industrial software product supporting process management.✓[43]✓+I labels2017Sustainability-aware MobilityCase study from the City of Amsterdam, exploring mobility as a service provided to the citizen to improve the safety, accessibility, air quality, quality of life, and attractiveness of Amsterdam.✓[44]✓2018Smart WorkScenarios to explore the implications of virtual mobility, i.e., substituting activities that require physical mobility with others that allow for virtual presence [45]✓[45]✓2020Food SecurityCase study to identify the sustainability and ethical concerns of designing context-sensitive ICT systems in low-resource environments. The general goal is to define a general good software design practice.✓[46, 47]✓+F,G2020Responsible Flight PlanningThe SAF Toolkit was applied in two workshops with experts from a multinational company specialized in operations- and flight planning products and software in the aviation sector. The goal was to identify sustainability opportunities and challenges for their flight planning software.✓-✓✓2021Information System (IS) Complexity and SustainabilityBased on the experience of IS professionals, this project identified a reusable taxonomy of concerns that are found to influence IS complexity and sustainability.✓[48]✓+C hierarchy✓2020-2022VU Winter SchoolsApplication of the SAF Toolkit to: four industrial cases in the domains energy provisioning, tourism, IT services, SDG reporting (2020); seven cases spanning from energy-efficient search engine, trustworthy open digital news, usable and sustainable opensource software platforms, to city-wide car sharing (2021); and four cases from paperless purchase receipts to digital mobile currency (2022).✓✓[49]✓+C clustering✓2020-2023Student Projects: Digitalization and Sustainability67 student group projects over four Master-level course editions, focused on digitally transforming aspects of various domains, from agriculture, education, retail, to entertainment, music festivals, arts.✓-✓+T✓2023Variability FeaturesIn an industrial case study the notion of variability features and variants (i.e., software features that can be implemented in different alternatives [41]) were introduced by applying and extending DMs. The SQ model was used to establish a measurement plan and monitor the impact of each emerged variant over time.✓[41]✓✓

4.2.2 Sustainability-Quality Model and its Instruments

The Conceptual Elements

In Figure2, the key elements of the SQ Model (coloured in blue) are encompassing quality attributes, dependencies among them, and associated metrics. Quality attributes, in turn, are characterized according to the sustainability dimension they belong to.

The SQ Model

The SQ Model comprises a collection of Quality Attributes (QAs) categorized into four Sustainability-Quality Dimensions (e.g., technical dimension includes security, environmental dimension includes energy efficiency). Recognizing the potential interdependencies among QAs and the multidimensional nature of software sustainability, such dependencies can be of two types:

(i) Inter-dimensional dependencies, linking a pair of QAs defined across different dimensions (e.g., robustness from the technical dimension positively influencing confidentiality in the social dimension).

(ii) Intra-dimensional dependencies, where a dependency emerges between two distinct QAs within the same dimension (e.g., within the technical dimension, security may depend on reliability).

Moreover, as our SQ model provides support to both identify design concerns and assess QAs of the software architecture, a set of metrics is used for measuring QAs. Similarly, metrics can be defined for sustainability-related requirements, represented as a class inheriting from class Design Concern.

The guiding Instruments

The SAF Toolkit further includes a number of instruments that provide guidance for the software architect to produce an SQ Model. These are the SQ Model Template and the Interdimensional Dependency Matrix Template, both described in the following.

The SQ Model Template provides a structured approach to assist software architects in creating their own specific SQ Model. It comprises three well-defined components:

• •

  Quality Attributes (QA): A curated list of QAs considered during the design or assessment of sustainability-aware software systems. Each attribute is accompanied by operational definitions delineating its intended purpose.

• •

  Direct Relationships to Sustainability Dimensions: Explicit mappings between QAs and overarching sustainability dimensions such as economic, environmental, social, and technical.

• •

  Metrics: Defined to offer a scale and method for measurement, the metrics encompass three types according to the standard ISO/IEC 25010:2011: internal metrics, external metrics, and quality-in-use metrics.

Overall, the template (see Figure6 for an example from [41]) is structured in a tabular format and consists of five fields: the name of the QA, its definition, the reference to the source of definition, the corresponding sustainability dimension, and a list of metrics.

Software architects have the flexibility to customize their SQ Model by either starting from a generic SQ Model that consolidates QA definitions from past projects and ISO/IEC 25010-2011 standards or by introducing new QAs not present in the generic model. They can also refine existing definitions and establish corresponding direct relationships.Figure 6 lists two QAs: Execution Time(ET) and Energy Efficiency (EE) that contribute to technical and environmental dimensions, respectively. Both QAs were identified as relevant of a specific sub-feature named variability in the context of Zahori. The metrics for ET and EE are variant specific, both are external metrics.

The SQ Model template transcends a mere static list of QAs related to sustainability dimensions; it serves as a dynamic toolset for informed decision-making and planning for the future. Through the construction of their specific SQ Models, architects can:

• •

  Identify QAs that are critical to achieving sustainability within specific contexts.

• •

  Incorporate sustainability considerations proactively into the software design process, hence fostering innovation and resilience in the face of evolving environmental, economic and social challenges.

The Interdimensional Dependency Matrix (DMatrix) Template constitutes another vital instrument within the SAF toolkit, designed to facilitate software architects in analyzing and managing dependencies among QAs of the SQ Model.The template allows architects to map out dependencies in a clear and organized manner, including instructions for its use and extension.The DMatrix is organized into rows and columns, each representing a specific QA within the sustainability dimensions D1 and D2, respectively. Three matrices, including technical-social, technical-economic, and technical-environmental were created and evaluated through a nichesourcing study, details can be found at [50].An example of the DMatrix template is shown inFigure7. The content of each cell of the DMatrix indicates the type of the dependency between the corresponding QAs:

• •

  Positive Dependency: A positive dependency between two QAs suggests that improvements or changes in one QA are likely to have positive effects on the other QA. For example, enhancing efficiency, that belongs to the technical dimension, may positively contribute to another QA of the social sustainability dimension like trust. A positive dependency is represented with the symbol “+”.

• •

  Negative Dependency: Conversely, a negative dependency indicates that changes or improvements in one QA may have adverse effects on the other QA. For example, increasing interoperability of the technical dimension may negatively affect another QA of the environmental dimension such as modifiability. A negative dependency is represented with the symbol “-”.

• •

  Indeterminate Dependency: Some cells may contain the value “I” indicating an indeterminate dependency between pairs of QAs from sustainability dimensions D1 and D2, requiring further contextual information to determine whether the dependency is positive or negative.

By examining the content of each cell, software architects can gain insights into the relationships between different QAs across sustainability dimensions. This understanding helps in decision-making processes and identifying potential trade-offs between sustainability concerns.

The DMatrix is instrumental in creating Decision Maps. The nature of dependencies identified in the DMatrix, whether positive, negative, or indeterminate, informs the type of effect between SQ concerns relevant for a project during the Decision Map creation process.

Reflections

The development and application of the SQ Model template and the DMatrix template have underscored the significance of integrating sustainability considerations into software architecture design. By integrating sustainability dimensions alongside QAs, software architects can make informed decisions to promote technical, environmental, social, and economic sustainability in software-intensive systems. Moreover, while we have introduced three matrices focusing on the technical dimension (i.e., technical-social, technical-environmental, technical-economic), validation efforts are currently in progress to explore dependencies among QAs across all four sustainability dimensions of SAF. This validation process is becoming increasingly feasible as software companies recognize the significance of addressing sustainability across all dimensions, rather than solely focusing on technical or environmental concerns.

However, there are areas that still demand further attention and development. One such area is the refinement of metrics and measurement techniques for evaluating sustainability aspects within software architecture. While existing standards provide a foundation, there is a need for more specific and tailored metrics to capture the unique challenges and opportunities of sustainable software development.Another area of improvement is related to the visualization the SQ Model. For instance, graph visualization could offer a more intuitive representation of the relationships among quality attributes from different sustainability dimensions, facilitating easier analysis and decision-making. Initial exploration in this direction has been presented in [51].Moreover, it is important to consider that graph-based models enable the persistence of software project data, supporting storage, retrieval, and manipulation for tracking changes over time. This lays the groundwork for future integration of artificial intelligence (AI) techniques, which can leverage graph data to identify patterns, such as common dependencies, or critical QRs.

4.3 The SAF Toolkit: Where we are going

As mentioned inSection4.1, we are working on two main extensions of the SAF Toolkit which we discuss in more details in Sections4.3.1 and 4.3.2. In addition, we explain how we plan to close the loop between architecture design and re-design, and actual measurement and monitoring of the corresponding system implementation, both driven by measures collected via the integration of the Green Lab (in Section4.3.3) and dashboards (in Section4.3.4).

4.3.1 Integration of KPIs

KPIs are widely used in business organizations to gauge their performance. The integration of the KPI framework [23] with the SAF Toolkit helps embody the notion of sustainability as a software quality in the operational structure of organizations. Figure8 shows the KPI Model, its elements and their relationships with the elements of the SAF Toolkit i.e., Sustainability Goal (from DM) and Metric (from SQ Model).

Business organizations have strategic high-level goals based on the organization’s vision and business model. These goals are based on Critical Success Factors (CSFs). A CSF defines the critical ingredients that are essential to concretizing and achieving the organizational goal. Achievement of CSFs assumes achievement of success for an organizational goal. This CSF operationalizes the KPI definition and helps motivate the choice of ‘what to measure?’ These measures can guide the quantification of metrics. These metrics can be used to measure the quality requirements in the SQ Model. With the integration of the KPI framework with the SAF Toolkit, the same metrics can be used to quantify the KPIs representing one or more quality requirements. Hence, the KPI is representative of the quality requirements as a function of the chosen metric(s).

The KPIs can help monitor the achievement of sustainability goals by using fitness functions to quantify the KPI and check target achievement. If and when KPI targets are not met, the KPI model provides feedback to the elements of the DM by stimulating appropriate actions that dictate the choice and execution of design decisions. This helps organizations create a synergy between their sustainability goals and concrete architecture design decisions through an action-driven approach. Supplemented by actions, organizations can take practical steps to improve the software architecture by integrating sustainability as a quality in a goal-focused and design-centric approach. This semantic structure aids in intentional, reusable, and value-focused data collection.

Reflections

A KPI is most useful when it is realized as a single value indicator with a threshold-based target. In this way, a single number can represent the achievement of success. This, however, is an optimistic assumption that has some practical obstacles, requiring a nuanced understanding of the system.

How to model the KPI functions?The biggest challenge with the KPI framework is choosing and developing the function to quantify the KPI by aggregating the chosen metrics. There is a need for knowledge-based evidence for the development of KPIs and their fitness functions. By using expert knowledge, regulatory support and mathematical modeling techniques, value-focused and relevant KPI functions could be developed.

How to evaluate the sustainability impact with dimensionally hybrid metrics?A KPI is a function of metrics that emerge from or impact one or more sustainability dimensions. We hypothesize that identifying and quantifying the sustainability impacts at the metric level can aid in quantifying the sustainability impact of the KPI across different sustainability dimensions. As identified by Hilty et al.[21], the representation of abstraction and gauging sustainability impact at different abstraction levels is complex. The metrics quantifying one KPI might have inter-dependencies. For instance, improving the value of a technical metric may lower the value of an environmental metric. Further, if a metric itself is dimensionally hybrid (i.e., intrinsically impacts multiple dimensions), then these dependencies become more complicated when viewed at a macro-level. Representing the impacts at this macro-level, where KPIs themselves could also be interdependent, can be complex. A systems thinking [52] approach is imperative for such impact analysis.

4.3.2 Integration of Software Architecture Descriptions

According to the Software Architecture ISO/IEC/IEEE 42010 Standard[40], a software architecture consists of multiple design decisions, as illustrated inFigure9. The SAF Toolkit incorporates this concept into its DM by integrating design concerns with design decisions. The DM aims to model sustainability concerns during the design phase, with these concerns manifesting as actual decisions in the software architecture at a later stage.

Our toolkit includes the concept of features, which must be distinguished from architectural design decisions. A feature is a desirable and externally observable software property [53]. As such it satisfies a requirement or represents a design decision[54]. Modeling these features allows architects to (re-)assess their decisions and their impact on sustainability during the design, or re-design.

The feature itself is realised_by one or more concrete architectural elements.

To provide a glimpse of the potential of the planned integration of software architecture descriptions into the SAF Toolkit, inFigure9 we show an example specific for the domain of service-oriented systems that we use in an educational context³³3One of the author is teaching a Master-level course on Service Oriented Design. There, software engineering competencies are taught on sustainability-focused projects[55]. (see dashed classes inFigure9). As such, a software service corresponds to a specific service-oriented system and its components.If explicitly modeled, such association would allow us to locate architecture elements in the software architecture and create the necessary mapping between features, decisions, and concerns. Only by introducing this mapping can we accurately evaluate the impact on sustainability of both the taken decisions and the specific architectural elements.

Reflections

It is worth noting that the newly introduced concept of aspects in accordance with the Software Architecture ISO/IEC/IEEE 42010:2022 Standard[40] can include concerns or non-functional properties (cf. Figure8).However, in the current version of the standard, aspects are only linked to properties of interest but are not explicitly linked to features resulting in a missing relationship to the architecture element. Therefore, the notion of aspects cannot yet be linked to the SAF Toolkit. We identify two challenges: (i) how can aspects be linked to features? (ii) how can we effectively link architecture elements as well as concerns? Tackling both challenges will help us to successfully provide architects with the necessary feedback about which of the taken decision(s) have an impact on which architecture element(s) and therefore on sustainability.

In the context of the KPI model, actions are triggered if a KPI does not reach its target. These actions are derived from the metrics quantifying the KPI. These metrics can be used to identify the quality requirement they represent and the associated design decisions. Hence, we can say that the actions can drive the choice of making a certain design decision if sustainability is not achieved over-time. However, as mentioned above, an underlying assumption is to be able to trace the measured impact to architecture elements.

4.3.3 Integration with the Green Lab

The Green Lab is the experimental platform developed at Vrije Universiteit Amsterdam for conducting empirical studies on the quality of software, with a special emphasis on its energy efficiency.At the time of writing, the Green Lab includes: (i) a cluster of 7 servers with different technical specifications for executing experiments targeting Cloud software, datacenters, and applications executing compute-intensive workloads, such as simulation, machine learning, and scientific computing, (ii) about 20 smartphones and 3 tablets for executing experiments on targeting mobile applications, (iii) 32 Raspberry Pi and 8 Arduino Nano boards to test embedded and IoT software, and (iv) other equipment including GPUs, educational robots, VR headsets, wearables.All devices in the Green Lab are equipped with state-of-the-art power monitors/profilers, both hardware and software (e.g., Monsoon power monitor, Watts Up Pro Meter, Intel RAPL, INA219).

The Green Lab also provides a set of open-source tools for conducting experiments on the energy efficiency of software. Among them, Experiment Runner[56] provides a Python library for collecting low-level measures from the power monitor/profiler, transforming them into higher-level metrics (e.g., energy), and persisting them according to an open file format (e.g., JSON), depending on the needs of the experiment.

We are planning to integrate (a custom version of) Experiment Runner in the SAF Toolkit. Specifically, by referring to Figure8, Experiment Runner will be in charge of (i) empirically collecting concrete measures from a running system/prototype (e.g., its average power consumption, CPU usage, exchanged HTTP messages) and (ii) making them available within the KPI model in the form of instances of the Measure concept. Collected measures will then be used to enrich the DM related to the running system.

Reflections

We expect that software architects using the DM enriched with the collected measures will have a better view of how the system is faring with respect to its KPIs, and thus the reasoning process of the architects will be more precise and grounded in the concrete characteristics of the system at hand. We are planning a campaign of user studies involving students and practitioners to either substantiate or disprove the just-mentioned expectation.

4.3.4 Integration with Monitoring Dashboard

In this paper, we primarily referred to the SAF Toolkit as a set of tools for supporting the design of software-intensive systems with sustainability as a first-class concern.However, the interest of software architects and decision makers is not confined to the construction and evolution of a software-intensive system, but rather it is critical to efficiently and reliably operate it throughout its complete life time[57, 58].This reflection is not new per se, but, with the advent of the microservices architectural style[59, 60] and the DevOps[61] paradigm, continuously operating a software-intensive system, promptly detecting relevant incidents (e.g., faults, performance issues), and diagnosing their potential impact at runtime has reached an unprecedented level of importance.

We believe that sustainability decisions are no exception and that the SAF Toolkit will be a valuable tool for continuously evaluating sustainability decisions (and their direct/indirect impact) at runtime.Central to this new line of research is the integration of the SAF Toolkit with a monitoring dashboard.In this context, monitoring consists of the collection of runtime metrics (e.g., energy consumption, memory/CPU usage, number of DB queries, average service latency) and their analysis to keep the overall quality of the system at acceptable levels[62].As of today, tens of monitoring tools exist for monitoring systems in the context of DevOps[63], such as Prometheus⁴⁴4https://prometheus.io or NetData⁵⁵5https://www.netdata.cloud.Monitoring platforms, e.g., Grafana⁶⁶6https://grafana.com provide dashboarding functionalities for querying the collected measures and graphically visualizing them via interactive tables, histograms, time series graphs, heatmaps, etc. Technically, the integration of the SAF Toolkit with a monitoring dashboard will consist of (i) a dedicated connector between the monitoring tool collecting measures at runtime and the dashboard, (ii) a widget for representing SAF instruments (e.g., decision maps) in the dashboard, and (iii) a set of mechanisms for mapping the tool-specific collected measures to elements of the KPI model, and in turn to the visualized SAF instrument.

Reflections

Using the SAF Toolkit in combination with a monitoring dashboard will be one of the first steps toward the integration of the architecture decision-making process and DevOps. As such, it will be valuable to expand the SAF Toolkit also with functionalities for helping architects promptly understanding the current situation of the system, identifying architecturally-relevant issues (e.g., the presence of performance bottlenecks or energy hotspots), and taking timely decisions about how to communicate and solve them. In this context, the usage of large language models tuned for suggesting architectural solutions/tactics in the context of the SAF Toolkit is a promising research direction that we are planning to pursue in future work.

5 Discussion and Conclusion

In this paper, we present the SAF Toolkit, a set of instruments aimed to support architects and design decision makers in modeling sustainability as a software quality property.

In our experience, we identified two main types of stakeholders as target users of the toolkit. The first are practicing architects and decision markers: in the many cases we could develop with them, we observed that architects consistently found the toolkit useful, both for novices that are new to sustainability in software architecting, and for experienced architects that are already knowledgeable of the relevance of sustainability-quality concerns in their projects but do not know how to extend their practices to apply it. The second type of target stakeholders are instructors that want to teach their students about sustainability in software engineering/architecture: we have years of experience in successfully doing that in master level courses, experience that we have translated in the supplementary on material[6], like small video tutorials (work in progress) and the checklist. Of course, practitioners willing to train themselves would find this beneficial, too.

When we described the current release of the SAF Toolkit (inSection4.2) and the extensions we planned (in Section4.3), we provided some reflections on the lessons learned and points for improvement.After over a decade of research in collaboration with industry and continuous improvements, the two instruments of DM and SQ Model in the current toolkit release are quite mature and consolidated into a powerful yet simple support for practitioners that may be both experienced and novices. Yet, we are missing the two extensions introducing instruments at design time (defining sound KPIs and linking sustainability elements with architecture elements other than features), and the two extensions that link the toolkit to a measurement environment (like our Green Lab) and a monitoring and reflection environment (like the integration with data collection and dashboarding).

We also care to go back to the definition of sustainability we provided inSection2, and our emphasis on the indivisible dimensions of sustainability-focus and of time. While time is represented in the DM by means of the mapping on impact levels (immediate, enabling and systemic), such representation is static as a DM is meant to visualize a model in a single point in time. Differently, we can truly assess if a software architecture addresses SQ concerns or is helping achieve a targeted sustainability goal only if we observe (i.e., measure and visualize) the trend distributed over a time period. Partially, the integration with the dashboard has the potential to represent measures (hence effects) over time. This however needs major research investments and is a main topic of our future research.

References

• \bibcommenthead
• European Commission and Directorate-General Energy [2023]European Commission,Directorate-General Energy:Reporting Requirements on the Energy Performance and Sustainability of Data Centres for the Energy Efficiency Directive. Task A Report, Options for a Reporting Scheme for Data Centres.Publications Office of the European Union,online(2023).https://data.europa.eu/doi/10.2833/304891
• Techieus [2023]Techieus:Cloud Computing Adoption: Statistics and Insights Across Sectors.https://techieus.com/cloud-computing-adoption-statistics-and-insights-across-sectors/.Accessed: 01-05-2024(2023)
• [3]Eurostat:Cloud computing: statistics on the use by enterprises.https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Cloud_computing_-_statistics_on_the_use_by_enterprises.Accessed: 01-05-2024
• Gozalo-Brizuela and Garrido-Merchan [2023]Gozalo-Brizuela, R.,Garrido-Merchan, E.C.:ChatGPT is not all you need. a state of the art review of large generative AI models.arXiv preprint arXiv:2301.04655(2023)
• Verdecchia etal. [2023]Verdecchia, R.,Sallou, J.,Cruz, L.:A systematic review of Green AI.Wiley interdisciplinary reviews. Data mining and knowledge discovery13(4),1507(2023)
• [6]The Sustainability Assessment Framework (SAF) Toolkit.https://github.com/S2-group/SAF-Toolkit.Online
• Calero etal. [2020]Calero, C.,Mancebo, J.,Garcia, F.,Moraga, M.A.,Berna, J.A.G.,Fernandez-Aleman, J.L.,Toval, A.:5Ws of green and sustainable software.Tsinghua Science and Technology25(3),401–414(2020)
• Peters etal. [2024]Peters, A.-K.,Capilla, R.,Coroamă, V.C.,Heldal, R.,Lago, P.,Leifler, O.,Moreira, A.,Fernandes, J.P.,Penzenstadler, B.,Porras, J.,Venters, C.C.:Sustainability in Computing Education: A Systematic Literature Review.ACM Transactions on Computing Education,3639060(2024)
• Calero and Piattini [2017]Calero, C.,Piattini, M.:Puzzling out Software Sustainability.Sustainable Computing: Informatics and Systems16,117–124(2017)
• Venters etal. [2023]Venters, C.C.,Capilla, R.,Nakagawa, E.Y.,Betz, S.,Penzenstadler, B.,Crick, T.,Brooks, I.:Sustainable software engineering: Reflections on advances in research and practice.Information and Software Technology,107316(2023)
• Lago [2023]Lago, P.:The Digital Society Is Already Here – Pity It Is ’Unsustainable’.In: Vermeulen, I. (ed.)Connected World - Insights from 100 Academics on How to Build Better Connections,pp. 55–58.VU University Press,Amsterdam(2023).https://vuuniversitypress.com/product/connected-world
• Becker etal. [2015]Becker, C.,Chitchyan, R.,Duboc, L.,Easterbrook, S.,Penzenstadler, B.,Seyff, N.,Venters, C.C.:Sustainability Design and Software: The Karlskrona Manifesto.In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering,pp. 467–476.IEEE,Florence, Italy(2015)
• Venters etal. [2014]Venters, C.C.,Lau, L.,Griffiths, M.K.,Holmes, V.,Ward, R.R.,Jay, C.,Dibsdale, C.E.,Xu, J.:The Blind Men and the Elephant: Towards an Empirical Evaluation Framework for Software Sustainability.Journal of Open Research Software2(1)(2014)
• Calero etal. [2021]Calero, C.,Moraga, M.Á.,Piattini, M.:Introduction to Software Sustainability.In: Calero, C.,Moraga, M.Á.,Piattini, M. (eds.)Software Sustainability,pp. 1–15.Springer International Publishing,Cham(2021)
• Andrikopoulos etal. [2022]Andrikopoulos, V.,Boza, R.-D.,Perales, C.,Lago, P.:Sustainability in Software Architecture: A Systematic Mapping Study.In: 2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA),pp. 426–433.IEEE,Gran Canaria, Spain(2022)
• Heldal etal. [2024]Heldal, R.,Nguyen, N.-T.,Moreira, A.,Lago, P.,Duboc, L.,Betz, S.,Coroamă, V.C.,Penzenstadler, B.,Porras, J.,Capilla, R.,Brooks, I.,Oyedeji, S.,Venters, C.C.:Sustainability competencies and skills in software engineering: An industry perspective.The Journal of systems and software211,111978(2024)
• Lago etal. [2015]Lago, P.,Koçak, S.A.,Crnkovic, I.,Penzenstadler, B.:Framing sustainability as a property of software quality.Communications of the ACM58(10),70–78(2015)
• Nandonde [2019]Nandonde, F.A.:A PESTLE analysis of international retailing in the East African Community.Global Business and Organizational Excellence38(4),54–61(2019)
• European Banking Authority [2022]European Banking Authority:Report on incorporating ESG risks in the supervision of investment firms.Technical Report EBA/REP/2022/26(2022)
• Duboc etal. [2020]Duboc, L.,Penzenstadler, B.,Porras, J.,AkinliKocak, S.,Betz, S.,Chitchyan, R.,Leifler, O.,Seyff, N.,Venters, C.C.:Requirements engineering for sustainability: An awareness framework for designing software systems for a better tomorrow.Requirements Engineering25(4),469–492(2020)
• Hilty and Aebischer [2015]Hilty, L.M.,Aebischer, B.:ICT for sustainability: An emerging research field.In: Advances in Intelligent Systems and Computing.Advances in intelligent systems and computing,pp. 3–36.Springer,Cham(2015)
• Parmenter [2015]Parmenter, D.:Key Performance Indicators: Developing, Implementing, and Using Winning KPIs.John Wiley & Sons, ???(2015)
• Fatima etal. [2023]Fatima, I.,Funke, M.,Lago, P.:From goals to actions: Providing guidance to software practitioners with kpis(2023)https://doi.org/10.36227/techrxiv.24057882.v1
• Staron etal. [2016]Staron, M.,Meding, W.,Niesel, K.,Abran, A.:A key performance indicator quality model and its industrial evaluation.In: 2016 Joint Conference of the International Workshop on Software Measurement and the International Conference on Software Process and Product Measurement (IWSM-MENSURA),pp. 170–179(2016).https://doi.org/10.1109/IWSM-Mensura.2016.033
• Staron etal. [2012]Staron, M.,Meding, W.,Palm, K.:Release readiness indicator for mature agile and lean software development projects.In: Lecture Notes in Business Information Processing.Lecture notes in business information processing,pp. 93–107.Springer,Berlin, Heidelberg(2012)
• Naumann etal. [2011]Naumann, S.,Dick, M.,Kern, E.,Johann, T.:The greensoft model: A reference model for green and sustainable software and its engineering.Sustainable Computing: Informatics and Systems1(4),294–304(2011)https://doi.org/10.1016/j.suscom.2011.06.004
• Cabot etal. [2009]Cabot, J.,Easterbrook, S.,Horkoff, J.,Lessard, L.,Liaskos, S.,Mazon, J.-N.:Integrating sustainability in decision-making processes: A modelling strategy.In: 2009 31st International Conference on Software Engineering - Companion Volume,pp. 207–210(2009).https://doi.org/10.1109/ICSE-COMPANION.2009.5070983
• Penzenstadler and Femmer [2013]Penzenstadler, B.,Femmer, H.:A generic model for sustainability with process- and product-specific instances.In: Proceedings of the 2013 Workshop on Green In/by Software Engineering.GIBSE ’13,pp. 3–8.Association for Computing Machinery,New York, NY, USA(2013).https://doi.org/10.1145/2451605.2451609 .https://doi.org/10.1145/2451605.2451609
• Saputri and Lee [2016]Saputri, T.R.D.,Lee, S.-W.:Incorporating sustainability design in requirements engineering process: A preliminary study.In: Lee, S.-W.,Nakatani, T. (eds.)Requirements Engineering Toward Sustainable World,pp. 53–67.Springer,Singapore(2016)
• Duboc etal. [2020]Duboc, L.,Penzenstadler, B.,Porras, J.,AkinliKocak, S.,Betz, S.,Chitchyan, R.,Leifler, O.,Seyff, N.,Venters, C.C.:Requirements engineering for sustainability: an awareness framework for designing software systems for a better tomorrow.Requirements Engineering25(4),469–492(2020)https://doi.org/10.1007/s00766-020-00336-y
• Penzenstadler etal. [2023]Penzenstadler, B.,Seyff, N.,Betz, S.,Duboc, L.,Porras, J.,Chitchyan, R.,Brooks, I.,Oyedeji, S.,Villela, K.B.,Venters, C.C.:Vision Paper: The Sustainability Awareness Framework (SusAF) as a De-Facto Standard?In: CEUR Workshop Proceedings,vol. 3378,p. 4(2023).CEUR Workshop Proceedings
• [32]Towards green AI‐based software systems: an architecture‐centric approach. GAISSA.https://gaissa.upc.edu/en.[Accessed 14-03-2024]
• Guldner etal. [2024]Guldner, A.,Bender, R.,Calero, C.,Fernando, G.S.,Funke, M.,Gröger, J.,Hilty, L.M.,Hörnschemeyer, J.,Hoffmann, G.-D.,Junger, D.,Kennes, T.,Kreten, S.,Lago, P.,Mai, F.,Malavolta, I.,Murach, J.,Obergöker, K.,Schmidt, B.,Tarara, A.,De Veaugh-Geiss, J.P.,Weber, S.,Westing, M.,Wohlgemuth, V.,Naumann, S.:Development and evaluation of a reference measurement model for assessing the resource and energy efficiency of software products and components—Green Software Measurement Model (GSMM).Future Generation Computer Systems155,402–418(2024)
• International Organization for Standardization [ISO] [2005]International Organization for Standardization [ISO]:Unified Modeling Language Specification.Technical Report ISO/IEC 19501:2005(E)(2005)
• deBoer and van Vliet [2009]Boer, R.C.,Vliet, H.:On the similarity between requirements and architecture.Journal of Systems and Software82(3),544–550(2009)https://doi.org/10.1016/j.jss.2008.11.185
• International Organization for Standardization [ISO] [2011]International Organization for Standardization [ISO]:Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models.Technical Report ISO/IEC 25010:2011(2011)
• Condori-Fernandez and Lago [2018]Condori-Fernandez, N.,Lago, P.:Characterizing the contribution of quality requirements to software sustainability.Journal of Systems and Software137,289–305(2018)
• Condori-Fernandez etal. [2020]Condori-Fernandez, N.,Lago, P.,Luaces, M.R.,Places, Á.S.:An Action Research for Improving the Sustainability Assessment Framework Instruments.Sustainability12(4),1682(2020)
• Betz etal. [2015]Betz, S.,Becker, C.,Chitchyan, R.,Duboc, L.,Easterbrook, S.M.,Penzenstadler, B.,Seyff, N.,Venters, C.C.:Sustainability Debt: A Metaphor to Support Sustainability Design Decisions.In: Fourth International Workshop on Requirements Engineering for Sustainable Systems (RE4SuSy),Ottawa, Canada(2015)
• International Organization for Standardization [ISO] [2022]International Organization for Standardization [ISO]:Systems and software engineering – Architecture description.Technical Report ISO/IEC/IEEE 42010:2022(November 2022)
• Funke etal. [2023]Funke, M.,Lago, P.,Verdecchia, R.:Variability Features: Extending Sustainability Decision Maps via an Industrial Case Study.In: 2023 IEEE 20th International Conference on Software Architecture Companion (ICSA-C),pp. 1–7.IEEE,L’Aquila, Italy(2023)
• Espana and Lago [2016]Espana, S.,Lago, P.:Software Sustainability Assessment (SoSA) exercise report.Technical report,Vrije Universiteit Amsterdam(May 2016).https://goo.gl/d9FYi9
• Verdecchia etal. [2017]Verdecchia, R.,Procaccianti, G.,Malavolta, I.,Lago, P.,Koedijk, J.:Estimating Energy Impact of Software Releases and Deployment Strategies: The KPMG Case Study.In: Nurcan, S.,Soffer, P.,Bajec, M.,Eder, J. (eds.)International Symposium on Empirical Software Engineering and Measurement (ESEM),pp. 257–266.Springer,Ljubljana, Slovenia(2017).https://doi.org/10.1109/ESEM.2017.39
• Niggebrugge etal. [2018]Niggebrugge, T.,Vos, S.,Lago, P.:The Sustainability of Mobility as a Service Solutions Evaluated through the Software Sustainability Assessment Method.Technical report,Vrije Universiteit Amsterdam(January 2018)
• Lago [2019]Lago, P.:Architecture Design Decision Maps for Software Sustainability.In: 41st International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS).ICSE,pp. 61–64.IEEE/ACM,Montréal, Canada(2019).http://dx.doi.org/10.1109/ICSE-SEIS.2019.00015
• Vos etal. [2020a]Vos, S.,Schaefers, H.,Bon, A.,Lago, P.:Sustainability and Ethics by Design: On the development of digital platforms in low-resource environments.Technical report,Vrije Universiteit Amsterdam(December 2020).https://research.vu.nl/files/137357217/ICT4FoodSec_1_.pdf
• Vos etal. [2020b]Vos, S.,Schaefers, H.,Lago, P.,Bon, A.:Towards Sustainability and Equality in Digital Development.In: Bandi, R.K., C R, R., Klein, S., Madon, S., Monteiro, E. (ed.)The Future of Digital Work: The Challenge of Inequality.IFIP AICT,vol. 601.Springer,Cham(2020).http://link.springer.com.vu-nl.idm.oclc.org/978-3-030-64697-4
• Bischoff etal. [2022]Bischoff, Y.,Wiel, R.,Hooff, B.,Lago, P.:A taxonomy about information systems complexity and sustainability.In: Wohlgemuth, V.,Naumann, S.,Behrens, G.,Arndt, H.-K. (eds.)Advances and New Trends in Environmental Informatics,pp. 17–33.Springer,Cham(2022)
• Lago etal. [2021]Lago, P.,Verdecchia, R.,Condori-Fernandez, N.,Rahmadian, E.,Sturm, J.,Nijnanten, T.,Bosma, R.,Debuysscher, C.,Ricardo, P.:Designing for Sustainability: Lessons Learned from Four Industrial Projects.In: Advances and New Trends in Environmental Informatics,pp. 3–18.Springer,Cham(2021).https://doi.org/10.1007/978-3-030-61969-5_1
• Condori-Fernandez etal. [2024]Condori-Fernandez, N.,Lago, P.,Catala, A.,Luaces, M.R.:Defining Interdimensional Dependencies of the Sustainability-Quality Model.Technical report,Vrije Universiteit Amsterdam(March 2024)
• Quispe-Cruz and Condori-Fernandez [2022]Quispe-Cruz, M.,Condori-Fernandez, N.:Extending the sustainability-quality model for supporting the design of persuasive software systems.In: Anais do XXV Congresso Ibero-Americano em Engenharia de Software,pp. 158–172.SBC,Porto Alegre, RS, Brasil(2022).https://doi.org/10.5753/cibse.2022.20970 .https://sol.sbc.org.br/index.php/cibse/article/view/20970
• Checkland [1999]Checkland, P.:Systems thinking. rethinking management information systems.Rethinking: Management information systems: An interdisciplinary perspective,44–56(1999)
• [53]Deckers, R.personal communication
• Apel and Kästner [2009]Apel, S.,Kästner, C.:An Overview of Feature-Oriented Software Development.Journal of Object Technology8(5),49–84(2009)
• CondoriFernandez and Lago [2018]CondoriFernandez, N.,Lago, P.:The Influence of Green Strategies Design onto Quality Requirements Prioritization.In: Requirements Engineering: Foundation for Software Quality,pp. 189–205.Springer, ???(2018).http://dx.doi.org/10.1007/978-3-319-77243-1_12
• [56]S2 Group:Experiment Runner.https://github.com/s2-group/experiment-runner/
• Fitzgerald and Stol [2014]Fitzgerald, B.,Stol, K.-J.:Continuous software engineering and beyond: trends and challenges.In: Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering,pp. 1–9(2014)
• van Hoorn etal. [2009]Hoorn, A.,Hasselbring, W.,Waller, J.,Ehlers, J.,Frey, S.,Kieselhorst, D.:Continuous monitoring of software services: Design and application of the kieker framework(2009)
• DiFrancesco etal. [2019]DiFrancesco, P.,Lago, P.,Malavolta, I.:Architecting with microservices: A systematic mapping study.Journal of Systems and Software150,77–97(2019)
• Waseem etal. [2021]Waseem, M.,Liang, P.,Shahin, M.,DiSalle, A.,Márquez, G.:Design, monitoring, and testing of microservices systems: The practitioners’ perspective.Journal of Systems and Software182,111061(2021)
• Jabbari etal. [2016]Jabbari, R.,Ali, N.,Petersen, K.,Tanveer, B.:What is DevOps? a systematic mapping study on definitions and practices.In: Proceedings of the Scientific Workshop Proceedings of XP2016.XP ’16 Workshops.Association for Computing Machinery,New York, NY, USA(2016).https://doi.org/10.1145/2962695.2962707 .https://doi.org/10.1145/2962695.2962707
• Ebert etal. [2016]Ebert, C.,Gallardo, G.,Hernantes, J.,Serrano, N.:DevOps.IEEE Software33(3),94–100(2016)
• Giamattei etal. [2024]Giamattei, L.,Guerriero, A.,Pietrantuono, R.,Russo, S.,Malavolta, I.,Islam, T.,Dinga, M.,Koziolek, A.,Singh, S.,Armbruster, M.,Gutierrez-Martinez, J.,Caro-Alvaro, S.,Rodriguez, D.,Weber, S.,Henss, J.,Fernandez Vogelin, E.,Simon Panojo, F.:Monitoring tools for devops and microservices: A systematic grey literature review.Journal of Systems and Software208,111906(2024)