Expensive objectives and constraints are key characteristics of real-world multi-objective optimization problems. In practice, they often occur jointly with inexpensive objectives and constraints.... Show moreExpensive objectives and constraints are key characteristics of real-world multi-objective optimization problems. In practice, they often occur jointly with inexpensive objectives and constraints. This paper presents the Inexpensive Objectives and Constraints Self-Adapting Multi-Objective Constraint Optimization algorithm that uses Radial Basis function Approximations (IOC-SAMO-COBRA) for such problems. This is motivated by the recently proposed Inexpensive Constraint Surrogate-Assisted Non-dominated Sorting Genetic Algorithm II (ICSA-NSGA-II). These algorithms and their counterparts that do not explicitly differentiate between expensive and inexpensive objectives and constraints are compared on 22 widely used test functions. The IOC-SAMOCOBRA algorithm finds significantly better (identical/worse) Pareto fronts in at least 78% (6%/16%) of all test problems compared to IC-SA-NSGA-II measured with both the hypervolume and Inverted Generational Distance+ performance metric. The empirical cumulative distribution functions confirm this advantage for both algorithm variants that exploit the inexpensive constraints. In addition, the proposed method is compared against state-of-the-art practices on a real-world cargo vessel design problem. On this 17-dimensional twoobjective practical problem, the proposed IOC-SAMO-COBRA outperforms SA-NSGA-II as well. From an algorithmic perspective, the comparison identifies specific strengths of both approaches and indicates how they should be hybridized to combine their best components. Show less
Lamers, C.H.C.B.; Vidal, R.; Belbachir, N.; Stein, N. van; Bäck, T.H.W.; Giampouras, P. 2023
We consider the problem of learning multiple tasks in a continual learning setting in which data from different tasks is presented to the learner in a streaming fashion. A key challenge in this... Show moreWe consider the problem of learning multiple tasks in a continual learning setting in which data from different tasks is presented to the learner in a streaming fashion. A key challenge in this setting is the so-called "catastrophic forgetting problem", in which the performance of the learner in an "old task" decreases when subsequently trained on a "new task". Existing continual learning methods, such as Averaged Gradient Episodic Memory (A-GEM) and Orthogonal Gradient Descent (OGD), address catastrophic forgetting by minimizing the loss for the current task without increasing the loss for previous tasks. However, these methods assume the learner knows when the task changes, which is unrealistic in practice. In this paper, we alleviate the need to provide the algorithm with information about task changes by using an online clustering-based approach on a dynamically updated finite pool of samples or gradients. We thereby successfully counteract catastrophic forgetting in one of the hardest settings, namely: domain-incremental learning, a setting for which the problem was previously unsolved. We showcase the benefits of our approach by applying these ideas to projection-based methods, such as A-GEM and OGD, which lead to task-agnostic versions of them. Experiments on real datasets demonstrate the effectiveness of the proposed strategy and its promising performance compared to state-of-the-art methods. Show less
Stein. N. van; Winter, R. de; Bäck, T.H.W.; Kononova, A. V. 2023
The conflict between computational budget and quality of found solutions is crucial when dealing with expensive black-box optimization problems from the industry. We show that through multi... Show moreThe conflict between computational budget and quality of found solutions is crucial when dealing with expensive black-box optimization problems from the industry. We show that through multi-objective parameter tuning of the Covariance Matrix Adaptation Evolution Strategy on benchmark functions different optimal algorithm configurations can be found for specific computational budgets and solution qualities. With the obtained Pareto front, tuned parameter sets are selected and transferred to a real-world optimization problem from vehicle dynamics, improving the solution quality and budget needed. The benchmark functions for tuning are selected based on their similarity to a real-world problem in terms of Exploratory Landscape Analysis features. Show less
Saturation is considered the state-of-the-art method for computing fixpoints with decision diagrams. We present a relatively simple decision diagram operation called REACH that also computes... Show moreSaturation is considered the state-of-the-art method for computing fixpoints with decision diagrams. We present a relatively simple decision diagram operation called REACH that also computes fixpoints. In contrast to saturation, it does not require a partitioning of the transition relation. We give sequential algorithms implementing the new operation for both binary and multi-valued decision diagrams, and moreover provide parallel counterparts. We implement these algorithms and experimentally compare their performance against saturation on 692 model checking benchmarks in different languages. The results show that the REACH operation often outperforms saturation, especially on transition relations with low locality. In a comparison between parallelized versions of REACH and saturation we find that REACH obtains comparable speedups up to 16 cores, although falls behind saturation at 64 cores. Finally, in a comparison with the state-of-the-art model checking tool ITS-tools we find that REACH outperforms ITS-tools on 29% of models, suggesting that REACH can be useful as a complementary method in an ensemble tool. Show less
Tannemaat, M.R.; Kefalas, M.; Geraedts, V.J.; Remijn-Nelissen, L.; Verschuuren, A.J.M.; Koch, M.; ... ; Bäck, T.H.W. 2023
OBJECTIVE\nMETHODS\nRESULTS\nCONCLUSIONS\nSIGNIFICANCE\nDistinguishing normal, neuropathic and myopathic electromyography (EMG) traces can be challenging. We aimed to create an automated time... Show moreOBJECTIVE\nMETHODS\nRESULTS\nCONCLUSIONS\nSIGNIFICANCE\nDistinguishing normal, neuropathic and myopathic electromyography (EMG) traces can be challenging. We aimed to create an automated time series classification algorithm.\nEMGs of healthy controls (HC, n = 25), patients with amyotrophic lateral sclerosis (ALS, n = 20) and inclusion body myositis (IBM, n = 20), were retrospectively selected based on longitudinal clinical follow-up data (ALS and HC) or muscle biopsy (IBM). A machine learning pipeline was applied based on 5-second EMG fragments of each muscle. Diagnostic yield expressed as area under the curve (AUC) of a receiver-operator characteristics curve, accuracy, sensitivity, and specificity were determined per muscle (muscle-level) and per patient (patient-level).\nDiagnostic yield of the classification ALS vs. HC was: AUC 0.834 ± 0.014 at muscle-level and 0.856 ± 0.009 at patient-level. For the classification HC vs. IBM, AUC was 0.744 ± 0.043 at muscle-level and 0.735 ± 0.029 at patient-level. For the classification ALS vs. IBM, AUC was 0.569 ± 0.024 at muscle-level and 0.689 ± 0.035 at patient-level.\nAn automated time series classification algorithm can distinguish EMGs from healthy individuals from those of patients with ALS with a high diagnostic yield. Using longer EMG fragments with different levels of muscle activation may improve performance.\nIn the future, machine learning algorithms may help improve the diagnostic accuracy of EMG examinations. Show less
Many state-of-the-art evolutionary algoritahms (EAs) can be categorized as sequential hybrid EAs, in which various EAs are sequentially executed. The timing to switch from one EA to another is... Show moreMany state-of-the-art evolutionary algoritahms (EAs) can be categorized as sequential hybrid EAs, in which various EAs are sequentially executed. The timing to switch from one EA to another is critical to the performance of the hybrid EA because the switching time determines the allocation of computational resources and thereby it helps balance exploration and exploitation. In this article, a framework for adaptive parameter control for hybrid EAs is proposed, in which the switching time is controlled by a learned agent rather than a manually designed scheme. First the framework is applied to an adaptive differential evolution algorithm, LSHADE, to control when to use the scheme to reduce the population. Then the framework is applied to the algorithm that won the CEC 2018 competition, i.e., the hybrid sampling evolution strategy (HSES), to control when to switch from the univariate sampling phase to the Covariance Matrix Adaptation Evolution Straategy phase. The agents for parameter control in LSHADE and HSES are trained by using Q-learning and deep Q-learning to obtain the learned algorithms Q-LSHADE and DQ-HSES. The results of experiments on the CEC 2014 and 2018 test suites show that the learned algorithms significantly outperform their counterparts and some state-of-the-art EAs. Show less
The algorithm selection problem is of paramount importance in achieving high-quality results while minimizing computational effort, especially when dealing with expensive black-box optimization... Show moreThe algorithm selection problem is of paramount importance in achieving high-quality results while minimizing computational effort, especially when dealing with expensive black-box optimization problems. In this paper, we address this challenge by using randomly generated artificial functions that mimic the landscape characteristics of the original problem while being inexpensive to evaluate. The similarity between the artificial function and the original problem is quantified using Exploratory Landscape Analysis. We demonstrate a significant performance improvement on five real-world vehicle dynamics problems by transferring the parameters of the Covariance Matrix Adaptation Evolution Strategy tuned to these artificial functions.We provide a complete set of simulated values of braking distance for fully enumerated 2D design spaces of all five real-world optimization problems. So, replication of our results and benchmarking directly on the real-world problems is possible. Beyond the scope of this paper, this data can be used as a benchmarking set for multi-objective optimization with up to five objectives. Show less
The performance of the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) is significantly affected by the selection of the specific CMA-ES variant and the parameter values used. Furthermore,... Show moreThe performance of the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) is significantly affected by the selection of the specific CMA-ES variant and the parameter values used. Furthermore, optimal CMA-ES parameter configurations vary across different problem landscapes, making the task of tuning CMA-ES to a specific optimization problem a challenging mixed-integer optimization problem. In recent years, several advanced algorithms have been developed to address this problem, including the Sequential Model-based Algorithm Configuration (SMAC) and the Tree-structured Parzen Estimator (TPE).In this study, we propose a novel approach for tuning CMA-ES by leveraging CMA-ES itself. Therefore, we combine the modular CMA-ES implementation with the margin extension to handle mixed-integer optimization problems. We show that CMA-ES can not only compete with SMAC and TPE but also outperform them in terms of wall clock time. Show less
A building spatial design (BSD) determines external and internal walls and ceilings of a building. The design space has a hierarchical structure, in which decisions on the existence or non... Show moreA building spatial design (BSD) determines external and internal walls and ceilings of a building. The design space has a hierarchical structure, in which decisions on the existence or non-existence of spatial components determine the existence of variables related to these spaces, such as sizing and angles. In the optimization of BSDs it is envisioned to optimize various performance indicators from multiple disciplines in concert, such as structural, functional, thermal, and daylight performance. Existing representations of design spaces suffer from severe limitations, such as only representing orthogonal designs or representing the structures in parametric superstructure, allowing only for limited design variations. This paper proposes prism nets - a new way of representing the search space of BSDs based on triangulations defining space filling collections of triangular prisms that can be combined via coloring parameters to spaces. Prism nets can accommodate for non-orthogonal designs and are flexible in terms of topological variations. We follow the guidelines for representation and operator design proposed in the framework of metric-based evolutionary algorithms. The main contribution of the paper is a detailed discussion of the search space representation and corresponding mutation operators. Moreover, a proof of concept example demonstrates the integration into multi-objective evolutionary algorithms and provides first results on a simple, but reproducible, benchmark problem. Show less
Zeiser, A.; Özcan, B.; Stein, N. van; Bäck, T.H.W. 2023
ObjectiveDistinguishing normal, neuropathic and myopathic electromyography (EMG) traces can be challenging. We aimed to create an automated time series classification algorithm.MethodsEMGs of... Show moreObjectiveDistinguishing normal, neuropathic and myopathic electromyography (EMG) traces can be challenging. We aimed to create an automated time series classification algorithm.MethodsEMGs of healthy controls (HC, n = 25), patients with amyotrophic lateral sclerosis (ALS, n = 20) and inclusion body myositis (IBM, n = 20), were retrospectively selected based on longitudinal clinical follow-up data (ALS and HC) or muscle biopsy (IBM). A machine learning pipeline was applied based on 5-second EMG fragments of each muscle. Diagnostic yield expressed as area under the curve (AUC) of a receiver-operator characteristics curve, accuracy, sensitivity, and specificity were determined per muscle (muscle-level) and per patient (patient-level).ResultsDiagnostic yield of the classification ALS vs. HC was: AUC 0.834 ± 0.014 at muscle-level and 0.856 ± 0.009 at patient-level. For the classification HC vs. IBM, AUC was 0.744 ± 0.043 at muscle-level and 0.735 ± 0.029 at patient-level. For the classification ALS vs. IBM, AUC was 0.569 ± 0.024 at muscle-level and 0.689 ± 0.035 at patient-level.ConclusionsAn automated time series classification algorithm can distinguish EMGs from healthy individuals from those of patients with ALS with a high diagnostic yield. Using longer EMG fragments with different levels of muscle activation may improve performance. Show less
Yarkoni, S.; Raponi, E.; Bäck, T.H.W.; Schmitt, S. 2022
Optimization problems with multiple objectives and many input variables inherit challenges from both large-scale optimization and multi-objective optimization. To solve the problems, decomposition... Show moreOptimization problems with multiple objectives and many input variables inherit challenges from both large-scale optimization and multi-objective optimization. To solve the problems, decomposition and transformation methods are frequently used. In this study, an improved control variable analysis is proposed based on dominance and diversity in Pareto optimization. Further, the decomposition method is used in a cooperative coevolution framework with orthogonal sampling mutation. The algorithm's performances are compared against the weighted optimization framework. The results show that the proposed decomposition method has much better accuracy compared to the traditional method. The results also show that the cooperative coevolution framework with a good grouping is very competitive. Additionally, the number of search directions in orthogonal sampling can be easily configured. A small number of search directions will reduce the search space greatly while also restricting the area that can be explored and vice versa. Show less
Real-world optimization scenarios under uncertainty and noise are typically handled with robust optimization techniques, which re-formulate the original optimization problem into a robust... Show moreReal-world optimization scenarios under uncertainty and noise are typically handled with robust optimization techniques, which re-formulate the original optimization problem into a robust counterpart, e.g., by taking an average of the function values over different perturbations to a specific input. Solving the robust counterpart instead of the original problem can significantly increase the associated computational cost, which is often overlooked in the literature to the best of our knowledge. Such an extra cost brought by robust optimization might depend on the problem landscape, the dimensionality, the severity of the uncertainty, and the formulation of the robust counterpart.This paper targets an empirical approach that evaluates and compares the computational cost brought by different robustness formulations in Kriging-based optimization on a wide combination (300 test cases) of problems, uncertainty levels, and dimensions. We mainly focus on the CPU time taken to find robust solutions, and choose five commonly-applied robustness formulations: `"mini-max robustness'', "mini-max regret robustness'', "expectation-based robustness'', ``dispersion-based robustness'', and "composite robustness'' respectively. We assess the empirical performance of these robustness formulations in terms of a fixed budget and a fixed target analysis, from which we find that "mini-max robustness'' is the most practical formulation w.r.t.~the associated computational cost. Show less
Manuel Proenca, H.; Grünwald, P.D.; Bäck, T.H.W.; Leeuwen, M. van 2022
We introduce the problem of robust subgroup discovery, i.e., finding a set of interpretable descriptions of subsets that 1) stand out with respect to one or more target attributes, 2) are... Show moreWe introduce the problem of robust subgroup discovery, i.e., finding a set of interpretable descriptions of subsets that 1) stand out with respect to one or more target attributes, 2) are statistically robust, and 3) non-redundant. Many attempts have been made to mine either locally robust subgroups or to tackle the pattern explosion, but we are the first to address both challenges at the same time from a global modelling perspective. First, we formulate the broad model class of subgroup lists, i.e., ordered sets of subgroups, for univariate and multivariate targets that can consist of nominal or numeric variables, including traditional top-1 subgroup discovery in its definition. This novel model class allows us to formalise the problem of optimal robust subgroup discovery using the Minimum Description Length (MDL) principle, where we resort to optimal Normalised Maximum Likelihood and Bayesian encodings for nominal and numeric targets, respectively. Second, finding optimal subgroup lists is NP-hard. Therefore, we propose SSD++, a greedy heuristic that finds good subgroup lists and guarantees that the most significant subgroup found according to the MDL criterion is added in each iteration. In fact, the greedy gain is shown to be equivalent to a Bayesian one-sample proportion, multinomial, or t-test between the subgroup and dataset marginal target distributions plus a multiple hypothesis testing penalty. Furthermore, we empirically show on 54 datasets that SSD++ outperforms previous subgroup discovery methods in terms of quality, generalisation on unseen data, and subgroup list size. Show less
