Showing posts with label research interest. Show all posts
Showing posts with label research interest. Show all posts

January 1, 2017

My PhD Thesis

I've finally got around to uploading a close-to-final draft of my PhD thesis (defended in early 2015). As the title suggests, the work is focused on describing several enhancements that can help multi-objective evolutionary algorithms (MOEAs) solve computationally-intensive optimization problems in a decent time frame. Large parts of the main novel scientific contributions of this thesis have been disseminated a priori in 7 peer-reviewed articles (3 journals and 4 conferences and edited volumes).

The practical scenarios I've tackled are all related to multi-objective optimizations of electrical drive designs and the very good results I've obtained on them also stem from a close  and very fruitful cooperation with the Linz Center of Mechatronics (LCM) and the Institute of Electrical Drives and Power Electronics (EAL) of the Johannes Kepler University Linz. Hopefully the version I've uploaded doesn't contain too many mistakes (scientific or language wise), but if you find any, please do let me know.

You can download a close-to-final draft of the paper by clicking here or from my Downloads box (Zavoianu Ciprian - PhD Thesis.pdf). The same version of the document can be previewed at the bottom of this post. For the official, final version of the document, please refer to the Johannes Kepler University LibraryIf you have any questions regarding the work, please don't hesitate to contact me.

For citations please use the following BibTeX reference:

@PhdThesis{Zavoianu2015PhD,
  author = {Alexandru-Ciprian Z\u{a}voianu},
  title = {Enhanced Evolutionary Algorithms for Solving Computationally-Intensive Multi-Objective Optimization Problems},
  school = {Johannes Kepler University Linz, Austria},
  year = {2015},
  month = {January}
}

July 1, 2016

Paper: Performance Comparison of Generational and Steady-State Asynchronous Multi-Objective Evolutionary Algorithms for Computationally-Intensive Problems

This journal article describes findings in one of the three main research foci I've investigated during my PhD research project. It is an improved and extended version of the conference manuscript prepared for ICAISC 2013Here is the abstract of the article:
In the last two decades, multi-objective evolutionary algorithms (MOEAs) have become ever more used in scientific and industrial decision support and decision making contexts the require an a posteriori articulation of preference. The present work is focused on a comparative analysis of the performance of two master-slave parallelization (MSP) methods, the canonical generational scheme and the steady-state asynchronous scheme. Both can be used to improve the convergence speed of multi-objective evolutionary algorithms that must use computationally-intensive fitness evaluation functions. Both previous and present experiments show that a correct choice for one or the other parallelization method can lead to substantial improvements with regard to the overall duration of the optimization process. Our main aim is to provide practitioners of MOEAs with a simple but effective method of deciding which MSP option is better given the particularities of the concrete optimization process. This in turn, would give the decision maker more time for articulating preferences (i.e., more flexibility). Our analysis is performed based on 15 well-known MOOP benchmark problems and two simulation-based industrial optimization process from the field of electrical drive design. For the first industrial MOOP, when comparing with a preliminary study, applying the steady state asynchronous MSP enables us to achieve an overall speedup (in terms of total wall-clock computation time) of ≈ 25%. For the second industrial MOOP, applying the steady-state MSP produces an improvement of ≈ 12%. We focus our study on two of the best known and most widely used MOEAs: the Non-dominated Sorting Genetic Algorithm II (NSGA-II) and the Strength Pareto Evolutionary Algorithm (SPEA2).
You can download the preprint version of the paper by clicking here or from my Downloads box (Performance Comparison of Generational and Steady-State Asynchronous Multi-Objective Evolutionary Algorithms for Computationally-Intensive Problems - KBS 2015.pdf). The same preprint version can be previewed at the bottom of this post. The final publication is available at elsevier.com.

For citations please use the following BibTeX reference:

@ARTICLE{Zavoianu2015KBS,
  author = {Alexandru-Ciprian Z\u{a}voianu and Edwin Lughofer and Werner Koppelst\"{a}tter and G\"{u}nther Weidenholzer and Wolfgang Amrhein and Erich Peter Klement},
  title = {Performance comparison of generational and steady-state asynchronous multi-objective evolutionary algorithms for computationally-intensive problems},
  journal = {Knowledge-Based Systems},

  note = {Computational Intelligence Applications for Data Science},
  year = {2015},
  volume = {87},
  pages = {47-60},
  doi = {10.1016/j.knosys.2015.05.029}
}


August 4, 2015

Paper: DECMO2 - A Robust Hybrid and Adaptive Multi-Objective Evolutionary Algorithm

This journal article also stems from my PhD research project - enhancing currently available Evolutionary Computation methods employed for solving computationally-intensive multi-objective optimization problems. The presented algorithm - DECMO2 - is an improved version of a method presented in one of our earlier papersHere is the abstract of the article:
We describe a hybrid and adaptive coevolutionary optimization method that can efficiently solve a wide range of multi-objective optimization problems (MOOPs) as it successfully combines positive traits from three main classes of multi-objective evolutionary algorithms (MOEAs): classical approaches that use Pareto-based selection for survival criteria, approaches that rely on differential evolution, and decomposition-based strategies. A key part of our hybrid evolutionary approach lies in the proposed fitness sharing mechanism that is able to smoothly transfer information between the coevolved subpopulations without negatively impacting the specific evolutionary process behavior that characterizes each subpopulation. The proposed MOEA also features an adaptive allocation of fitness evaluations between the coevolved populations in order to increase robustness and favor the evolutionary search strategy that proves more successful for solving the MOOP at hand. Apart from the new evolutionary algorithm, this paper also contains the description of a new hypervolume and racing-based methodology aimed at providing practitioners from the field of multi-objective optimization with a simple means of analyzing/reporting the general comparative run-time performance of multi-objective optimization algorithms over large problem sets.
You can download the preprint version of the paper by clicking here or from my Downloads box (DECMO2 - A Robust Hybrid and Adaptive Multi-Objective Evolutionary Algorithm - SOCO 2014.pdf). The same preprint version can be previewed at the bottom of this post. The final publication is available at Springer via http://dx.doi.org/10.1007/s00500-014-1308-7.

For citations please use the following BibTeX reference:

@ARTICLE{Zavoianu2014SOCO,
  author = {Alexandru-Ciprian Z\u{a}voianu and Edwin Lughofer and Gerd Bramerdorfer and Wolfgang Amrhein and Erich Peter Klement},
  title = {{DECMO2}: a robust hybrid and adaptive multi-objective evolutionary algorithm},
  journal = {Soft Computing},
  year = {2014},
  volume = {19},
  number = {12},
  pages = {3551-3569},
  doi = {10.1007/s00500-014-1308-7}
}

March 17, 2015

Paper: An Effective Ensemble-Based Method for Creating On-the-Fly Surrogate Fitness Functions for Multi-Objective Evolutionary Algorithms - SYNASC 2013

This paper was prepared for the SYNASC 2013 conference and it is related to my current research project that has the general aim of enhancing currently available Evolutionary Computation methods employed for the multi-objective optimization of problems that rely on a very time-intensive fitness evaluation functionsHere is the abstract of the article:
The task of designing electrical drives is a multi-objective optimization problem (MOOP) that remains very slow even when using state-of-the-art approaches like particle swarm optimization and evolutionary algorithms because the fitness function used to assess the quality of a proposed design is based on time-intensive finite element (FE) simulations. One straightforward solution is to replace the original FE-based fitness function with a much faster-to-evaluate surrogate. In our particular case each optimization scenario poses rather unique challenges (i.e., goals and constraints) and the surrogate models need to be constructed on-the-fly, automatically, during the run of the evolutionary algorithm. In the present research, using three industrial MOOPs, we investigated several approaches for creating such surrogate models and discovered that a strategy that uses ensembles of multi-layer perceptron neural networks and Pareto-trimmed training sets is able to produce very high-quality surrogate models in a relatively short time interval.
You can download a preliminary version of the paper by clicking here or from my Downloads box (On the Performance of Master-Slave Parallelization Methods for MOEAs - ICAISC 2013.pdf). The same preliminary draft of the document can be previewed at the bottom of this post. The original publication is available at the IEEE Xplore website.

For citations please use the following BibTeX reference:

@INPROCEEDINGS{Zuavoianu2013SYNASC,
  author = {Alexandru-Ciprian Z\u{a}voianu and Edwin Lughofer and Gerd Bramerdorfer and Wolfgang Amrhein and Erich Peter Klement},
  title = {An Effective Ensemble-Based Method for Creating On-the-Fly Surrogate Fitness Functions for Multi-Objective Evolutionary Algorithms},
  booktitle = {Proceedings of the 15th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC 2013)},
  year = {2013},
  pages = {235-248},
  publisher = {IEEE Computer Society},
}

March 1, 2014

Paper: Efficient Multi-Objective Optimization Using 2-Population Cooperative Coevolution - EUROCAST 2013

This paper was prepared for the EUROCAST 2013 conference and contains a proof-of-concept for DECMO - a simple coevolutionary multi-objective optimization algorithm that, on several problems is successfully able to adopt the search behavior of the most successful strategy it incorporates. This line of research was continued and fully fleshed out in DECMO2 - a subsequent algorithm I developed for my PhD. Here is the abstract our EUROCAST 2013 submission:
We propose a 2-population cooperative coevolutionary optimization method that can efficiently solve multi-objective optimization problems as it successfully combines positive traits from classic multi-objective evolutionary algorithms and from newer optimization approaches that explore the concept of differential evolution. A key part of the algorithm lies in the proposed dual fitness sharing mechanism that is able to smoothly transfer information between the two coevolved populations without negatively impacting the independent evolutionary process behavior that characterizes each population.
You can download a preliminary version of the paper by clicking here or from my Downloads box (Efficient Multi-Objective Optimization Using 2-Population Cooperative Coevolution - EUROCAST 2013.pdf). The same preliminary draft of the document can be previewed at the bottom of this post. The original publication is available at www.springerlink.com.

For citations please use the following BibTeX reference:

@INCOLLECTION{Alexandru-CiprianZavoianu2013,
  author = {Alexandru-Ciprian Z\u{a}voianu and Edwin Lughofer and Wolfgang Amrhein and Erich Peter Klement},
  title = {Efficient Multi-Objective Optimization Using 2-Population Cooperative Coevolution},
  booktitle = {Computer Aided Systems Theory - EUROCAST 2013},
  publisher = {Springer Berlin / Heidelberg},
  year = {2013},
  editor = {Moreno-Diaz, Roberto and Pichler, Franz and Quesada-Arencibia, Alexis},
  volume = {8111},
  series = {Lecture Notes in Computer Science},
  pages = {251-258},
  doi = {10.1007/978-3-642-53856-8_32},
}


October 7, 2013

Paper: Hybridization of Multi-Objective Evolutionary Algorithms and Artificial Neural Networks for Optimizing the Performance of Electrical Drives - EAAI

A big part of my current PhD work at the JKU-Department of Knowledge-Based Mathematical Systems is related to the dissemination of our current research (i.e., writing scientific articles). That's why most of my recent / forthcoming posts are / will be about "papers".

The work presented in this post is a revised and extended (journal) version of one of the earlier papers written in collaboration with our partners from the Institute for Electrical Drives and Power Electronics of the Johannes Kepler University, Linz. The aim of the article is to describe a surrogate-based enhancement that can help to significantly speed-up a multi-objective evolutionary algorithm that requires an extremely time-intensive fitness evaluation function. Here is the abstract of the article:
Performance optimization of electrical drives implies a lot of degrees of freedom in the variation of design parameters, which in turn makes the process overly complex and sometimes impossible to handle for classical analytical optimization approaches. This, and the fact that multiple non-independent design parameter have to be optimized synchronously, makes a soft computing approach based on multi-objective evolutionary algorithms (MOEAs) a feasible alternative. In this paper, we describe the application of the well known Non-dominated Sorting Genetic Algorithm II (NSGA-II) in order to obtain high-quality Pareto-optimal solutions for three optimization scenarios. The nature of these scenarios requires the usage of fitness evaluation functions that rely on very time-intensive finite element (FE) simulations. The key and novel aspect of our optimization procedure is the on-the-fly automated creation of highly accurate and stable surrogate fitness functions based on artificial neural networks (ANNs). We employ these surrogate fitness functions in the middle and end parts of the NSGA-II run (=> hybridization) in order to significantly reduce the very high computational effort required by the optimization process. The results show that by using this hybrid optimization procedure, the computation time of a single optimization run can be reduced by 46% to 72% while achieving Pareto-optimal solution sets with similar, or even slightly better, quality as those obtained when conducting NSGA-II runs that use FE simulations over the whole run-time of the optimization process.
You can download the preprint version of the paper by clicking here or from my Downloads box (Hybridization of Multi-Objective Evolutionary Algorithms and Artificial Neural Networks for Optimizing the Performance of Electrical Drives - EAAI 2013.pdf). The same preprint version can be previewed at the bottom of this post. The original publication is available at elsevier.com.

For citations please use the following BibTeX reference:


@ARTICLE{Zavoianu2013EAAI,
  author = {Alexandru-Ciprian Z\u{a}voianu and Gerd Bramerdorfer and Edwin Lughofer and Siegfried Silber and Wolfgang Amrhein and Erich Peter Klement},
  title = {Hybridization of Multi-Objective Evolutionary Algorithms and Artificial Neural Networks for Optimizing the Performance of Electrical Drives},
  journal = {Engineering Applications of Artificial Intelligence},
  year = {2013},
  volume = {26},
  pages = {1781-1794},
  number = {8},
  doi = {10.1016/j.engappai.2013.06.002}
}

June 13, 2013

Paper: On the Performance of Master-Slave Parallelization Methods for Multi-Objective Evolutionary Algorithms - ICAISC 2013

This paper was prepared for the ICAISC 2013 conference and it is related to my current research project that has the general aim of enhancing currently available Evolutionary Computation methods employed for the multi-objective optimization of problems that rely on a very time-intensive fitness evaluation functionsHere is the abstract of the article:
This paper is focused on a comparative analysis of the performance of two master-slave parallelization methods, the basic generational scheme and the steady-state asynchronous scheme. Both can be used to improve the convergence speed of multi-objective evolutionary algorithms (MOEAs) that rely on time-intensive fitness evaluation functions. The importance of this work stems from the fact that a correct choice for one or the other parallelization method can lead to considerable speed improvements with regards to the overall duration of the optimization. Our main aim is to provide practitioners of MOEAs with a simple but effective method of deciding which master-slave parallelization option is better when dealing with a time-constrained optimization process.
You can download a preliminary version of the paper by clicking here or from my Downloads box (On the Performance of Master-Slave Parallelization Methods for MOEAs - ICAISC 2013.pdf). The same preliminary draft of the document can be previewed at the bottom of this post. The original publication is available at www.springerlink.com.

For citations please use the following BibTeX reference:

@INCOLLECTION{Zavoianu2013ICAISC,
  author = {Alexandru-Ciprian Z\u{a}voianu and Edwin Lughofer and Werner Koppelst\"{a}tter and G\"{u}nther Weidenholzer and Wolfgang Amrhein and Erich Peter Klement},
  title = {On the Performance of Master-Slave Parallelization Methods for Multi-Objective Evolutionary Algorithms},
  booktitle = {Artificial Intelligence and Soft Computing},
  publisher = {Springer Berlin Heidelberg},
  year = {2013},
  editor = {Laszek Rutkowski et al.},
  volume = {7895},
  series = {Lecture Notes in Artificial Intelligence},
  pages = {122-134},
  doi = {10.1007/978-3-642-38610-7_12}
}

September 5, 2012

Paper: A Hybrid Soft Computing Approach for Optimizing Design Parameters of Electrical Drives - SOCO 2012

This paper was prepared for the SOCO 2012 conference and it provides some insight into my current research project that has the general aim of enhancing currently available Evolutionary Computation methods employed for the optimization of design parameters used in the prototyping process of electrical drives. The present paper is particularly related to this earlier post regarding the treatment of expensive fitness evaluation functions in Evolutionary Computation. Here is the abstract of the article:
In this paper, we are applying a hybrid soft computing approach for optimizing the performance of electrical drives where many degrees of freedom are allowed in the variation of design parameters. The hybrid nature of our approach originates from the application of multi-objective evolutionary algorithms (MOEAs) to solve the complex optimization problems combined with the integration of non-linear mappings between design and target parameters. These mappings are based on artificial neural networks (ANNs) and they are used for the fitness evaluation of individuals (design parameter vectors). The mappings substitute very time-intensive finite element simulations during a large part of the optimization run. Empirical results show that this approach finally reduces the computation time for single runs from a few days to several hours while achieving Pareto fronts with a similar high quality.
You can download a preliminary version of the paper by clicking here or from my Downloads box (A Hybrid Soft Computing Approach for Optimizing Design Parameters of Electrical Drives - SOCO 2012.pdf). The same preliminary draft of the document can be previewed at the bottom of this post. The original publication is available at www.springerlink.com.

For citations please use the following BibTeX reference:

@INCOLLECTION{ Alexandru-CiprianZavoianu2013,
  author = {Alexandru-Ciprian Z\u{a}voianu and Gerd Bramerdorfer and Edwin Lughofer and Siegfried Silber and Wolfgang Amrhein and Erich Peter Klement},
  title = {A Hybrid Soft Computing Approach for Optimizing Design Parameters of Electrical Drives},
  booktitle = {Soft Computing Models in Industrial and Environmental Applications},
  publisher = {Springer Berlin Heidelberg},
  year = {2013},
  editor = {V\'{a}clav Sn\'{a}\u{s}el and Ajith Abraham and Emilio S. Corchado},
  volume = {188},
  series = {Advances in Intelligent Systems and Computing},
  pages = {347-358},
  doi = {10.1007/978-3-642-32922-7_36}
}

February 2, 2012

Paper: Improving the Parsimony of Regression Models for an Enhanced Genetic Programming Process - EUROCAST 2011

This paper was prepared for the EUROCAST 2011 conference and it basically contains a short summary of the work I have carried out for my Master Thesis Project. Here is the abstract of the article:
This research is focused on reducing the average size of the solutions generated by an enhanced GP process without affecting the high predictive accuracy this method exhibits when being applied on a complex, industry proposed, regression problem. As such, the effects the GP enhancements have on bloat have been studied and, finally, a bloat control system based on dynamic depth limiting (DDL) and iterated tournament pruning (ITP) was designed. The resulting bloat control system is able to improve by approx. 40% the average GP solution parsimony without impacting average solution accuracy.
You can download a preliminary version of the paper by clicking here or from my Downloads box (Improving the Parsimony of Regression Models for an Enhanced GP Process - EUROCAST 2011.pdf). The same preliminary draft of the document can be previewed at the bottom of this post. The original publication is available at www.springerlink.com.

For citations please use the following BibTeX reference:

@INCOLLECTION{Alexandru-CiprianZavoianu2012,
  author = {Alexandru-Ciprian Z\u{a}voianu and Gabriel Kronberger and Michael Kommenda and Daniela Zaharie and Michael Affenzeller},
  title = {Improving the Parsimony of Regression Models for an Enhanced Genetic Programming Process},
  booktitle = {Computer Aided Systems Theory - EUROCAST 2011},
  publisher = {Springer Berlin / Heidelberg},
  year = {2012},
  editor = {Moreno-Diaz, Roberto and Pichler, Franz and Quesada-Arencibia, Alexis},
  volume = {6927},
  series = {Lecture Notes in Computer Science},
  pages = {264-271},
  doi = {10.1007/978-3-642-27549-4_34},
}


July 9, 2011

Tips & Tricks: Handling expensive evaluation functions in Evolutionary Computation

Evolutionary algorithms (EA) have become very popular for solving complex optimization / data based modeling problems as, generally, they are fairly robust, easy to use and able to produce high quality results.

A common problem in real-word applications is the fact that the evaluation/fitness function employed by this type of algorithms is quite expensive (time-wise or even financially) to evaluate. As such, when taking into consideration the restrictions imposed on the running time of the EA, only a limited amount of fitness evaluations can be performed and this usually has a huge impact on the quality of the obtained solutions.

The approaches that have been proposed by scientific literature to deal with this problem roughly fall within two main categories:
  • surrogate approximation - construct a new (faster to evaluate) expression of the objective function based on previous input to output mappings obtained using the original (expensive) evaluation function.
  • evolutionary approximation - reduce the number of expensive fitness evaluations performed during the run by trying to estimate the fitness of an individual without using the costly evaluation function by combining information from other similar individuals that have already been evaluated and, if available, information regarding parameter sensitivity.
Two of the most common methods for surrogate approximation are based on neural networks (NNs) or support vector machines (SVMs). The idea behind the approach is quite simple: when having a sufficient number of individual fitness evaluations (i.e. sampling data), try to train a NN or a SVM model that can approximate reasonably well the actual expensive fitness evaluation function. The quality of the approximation depends a lot on the complexity and roughness of the search space, on the amount of sampling data and of course on the choice of NN or SVM kernel. After the surrogate NN or SVM model has been trained we can use it to perform a lot of inexpensive but relatively accurate fitness evaluations during parts, or even during the entire run of the EA.

Evolutionary approximation is based on the fact that, for most search spaces, two individuals that share many common features also tend to have a similar fitness value. For example, if in an EA's individuals are coded by a real parameter vector containing 15 values and one individual has 14 values that match exactly the values of an individual that has been evaluate 3 generations before and the 15 value is only slightly different, we can approximate the fitness of the current individual with that of its ancestor. This is a very brute approximation, that can be extended and refined (i.e. using advanced similarity metrics, interpolation, variable correlation information, local mappings, etc) according to the concrete problem at hand.

Hybrid strategies are also worth mentioning. One idea could be to use the surrogate models especially in the middle stages of the EA run and then to switch back to the original fitness evaluation function at the the very end stages of the run, when precision is of the utmost importance.  

December 5, 2010

My Master's Thesis

As promised in my previous post regarding code bloating in GP, here is my Master's Thesis. Hopefully the version I've uploaded doesn't contain too many mistakes (scientific or language wise), but if you find any, please do let me know.

You can download a close-to-final draft of the paper by clicking here or from my Downloads box (Zavoianu Ciprian - Master's Thesis.pdf). The same version of the document can be previewed at the bottom of this post. For the official, final version of the document, please refer to the Johannes Kepler University Library. If you have any questions regarding it, please don't hesitate to contact me.

For citations please use the following BibTeX reference or refer to the conference article that summarizes this work:

@MASTERSTHESIS{Zavoianu2010,
  author = {Alexandru-Ciprian Z\u{a}voianu},
  title = {Towards solution parsimony in an enhanced genetic 
    programming process},
  school = {International School Informatics: Engineering \& 
    Management, ISI-Hagenberg, Johannes Kepler University, Linz},
  year = {2010}
}

November 20, 2010

Code Bloating in Genetic Programming

In this post I'll try to present some information regarding the main subject of my Master's Thesis which is entitled "Towards Solution Parsimony in an Enhanced Genetic Programming Process" (sounds a bit bombastic, I know :P). Work on this paper has been carried out at the Heuristic and Evolutionary Algorithms Laboratory (HEAL) in the Hagenberg Softwarepark, Austria. My research was coordinated by Prof. Dr. Michael Affenzeller and Prof. Dr. Daniela Zaharie and was sponsored by the SPRERS project and ISI Hagenberg.

In order to present what I've worked on for the better part of one year, I'll start with a concise wiki definition of what genetic programming (GP) is:
In artificial intelligence, genetic programming (GP) is an evolutionary algorithm-based methodology, inspired by biological evolution, to find computer programs that perform a user-defined task. It is a specialization of genetic algorithms (GA) where each individual is a computer program.
As the user-defined task can vary from detecting military equipment in  satellite recconaissance  photos to designing analogue circuits, to solving symbolic regression problems or to evolving competitive AI players, it is easy to see that GP is a domain-independent method that can be used either as an automatic programming tool, a machine learning tool or a problem solving engine.

One of the main problems GP practitioners have always been confronted with is a phenomenon called code bloat: the average size of the GP evolved programs starts to grow at a very rapid pace after a certain number of generations but this increase in overall program complexity (size) is not matched by any corresponding increase in program accuracy. In time, bloat brings the entire evolutionary process to a grinding halt

The problem is not a trivial one as there are several studies that tie code bloating to the driving force behind GP and most evolutionary methods: the search for fitness

The objective of my Master Thesis was to enhance the general interpretability of GP based symbolic regression models produced by HeuristicLab (the heuristic optimization framework developed by HEAL). The approach for achieving this objective focused on understanding and reducing/controlling the bloating phenomenon that affected the enhanced GP process implemented in HeuristicLab. For more details please consult the full paper which is presented in this post.

GP has become a quite intense field of the study in the last couple of years and there is a wealth of (high quality) scientific papers regarding various aspects of this evolutionary method. For those that are very interested in GP, a good start is an excellent free online book about GP written by Ricardo Poli and William Langdon, two of the most well known researchers in the field, alongside John Koza of course.

January 19, 2009

Project: Bandwidth reduction

This post is a follow-up to the problem of bandwidth reduction presented earlier. I've done a small Delphi application that can generate sparse matrices of various sizes and densities (the density of a sparse matrix is the percent of non-zero elements is contains) and that features 3 (well, actually more like 2) bandwidth reduction algorithms. The first algorithm is the original Cuthill-McKee algorithm, the second one is the Reverse Cuthill-Mckee algorithm (both of which have been covered in detail) and the third algorithm I've decided to implement is a genetic algorithm.
.
I must say that I was very curious to see how a genetic algorithm would perform on this problem as the bandwidth reduction problem is basically a combinatorial problem. As it turns out there is a really nice study performed by A. Lim, B. Rodrigues and F. Xiao that covers this exact topic. The three researchers carry on the work of R. Marti and V. Campos in trying to use a metaheurisc approach in order to obtain better results than the classic bandwidth reduction algorithms (RCM and GPS).
.
Although genetic algorithms perform very well in global search, some researchers argue that they are less suitable for highly tuned search. As such, Lim, Rodrigues and Xiao combined their genetic algorithm with a hill climbing approach. The resulting hybrid (memetic) algorithm performs very well and generally produces solutions of better quality than the Reverse Cuthill-McKee algorithm.
.
My implementations of the CutHill-McKee algorithm and of the genetic algorithm are by no means optimal! (the genetic algorithm is also slightly modified). Hence, both algorithms perform relatively poor (in terms of speed) compared with their creators' estimations. However, the main purpose of this project was to show that in the field of NP-complete problem solving, metaheuristic approaches, although considered rather slow, can be expected to produce very high quality results.
.
You cand download a fully working version of the application (source code included) by clicking here or from the Downloads box (BandwidthReduction.zip). Here are two screenshots
:


And here is an example of how RCM and the hybrid metaheuristics-based algorithm performed on a 150x150 randomly generated simetric sparse matrix with a density of 3.5% (788 non-zero elements):

1. The initial sparse matrix. (bandwidth = 142)

 
2. The solution obtained using RCM. (bandwidth = 81)

 
3. The solution obtaind using the hybrid genetic + hill climbing algorithm. (bandwidth = 65). This solution was obtained after 15 generations using a mutation rate of 40%.


Application platform: Win32

January 15, 2009

Tutorial: Bandwidth reduction - The CutHill-McKee Algorithm

1. The problem

I've recently come across an interesting NP-complete problem :D. It's the bandwidth reduction problem, also known in the field of sparse matrix applications as the bandwidth minimization problem (or BMP in short):
For a given simetric sparse matrix, M(nxn), the problem is to reduce its bandwidth B by permuting rows and columns such as to move all the nonzero elements of M in a band as close as possible to the diagonal.
In other words, the problem consists of transforming through successive row and column permutations this (8x8 input matrix):


into, possibly, this (8x8 output matrix):


Now here's an interesting thing regarding this problem. If you consider the given sparse matrix (the input matrix) and the resulting matrix (the output matrix) as adjacency matrices for some given graphs, this is what you end up with:
.
The graph associated with the input matrix.


The graph associated with the output matrix.


As you can easily see, the two graph structures are identical, the only thing that is different is the node (vertex) labeling. In other words the bandwidth reduction problem can also be viewed as a graph labeling problem:
Find the node labeling that minimizes the bandwidth B of the adjacency matrix of the graph G(n) , where we can formally define: B=max|Li-Lj|, i,j=1..n and Li is the label of node i, Lj is the label of node j and nodes i and j are adjacent or neighbours (i.e., there is an edge in graph G(n) connecting nodes i and j).

2. The solution

Back in the 60's and 70's many graph algorithms were proposed for "solving" the bandwidth reduction problem. I wrote "solving" beacause the problem is after all NP-complete and none of the algorithms described claimed to find the exact solution(s) regardless of the input matrix, but they were more or less succesfull in finding a relatively good solution in a resonable amount of time (the so called heuristic approach to NP-complete problem solving).
.
Initially the most widely used of these heuristics was the Reverse Cuthill-McKee algorithm (RCM), a modification by Alan George of the original algorithm developed by CutHill and McKee in 1969. In 1976 the GPS algorithm (named after its developers Gibbs, Poole and Stockmeyer) was proposed. On average GPS and RCM find solutions of relative equal quality, but GPS is up to ten times as fast.

One last thing I should remind you before I continue with the description of the RCM algorithm: the degree of a node in a graph is the number of nodes adjacent to it. So here is the RCM algorithm for a given graph G(n).

Step 0: Prepare an empty queue Q and an empty result array R.
Step 1: Select the node in G(n) with the lowest degree (ties are broken arbitrarily) that hasn't previously been inserted in the result array. Let us name it P (for Parent).
Step 2: Add P in the first free position of R.
Step 3: Add to the queue all the nodes adjacent with P in the increasing order of their degree.
Step 4.1: Extract the first node from the queue and examine it. Let us name it C (for Child).
Step 4.2: If C hasn't previously been inserted in R, add it in the first free position and add to Q all the neighbours of C that are not in R in the increasing order of their degree.
Step 5: If Q is not empty repeat from Step 4.1 .
Step 6: If there are unexplored nodes (the graph is not connected) repeat from Step 1 .
Step 7: Reverse the order of the elements in R. Element R[i] is swaped with element R[n+1-i].

The result array will be interpreted like this: R[L] = i means that the new label of node i (the one that had the initial label of i) will be L.

As you can see steps 1-6 describe nothing but a classic Breadth First Search (BFS) with the minor modification that the nodes are explored in the increasing order of their degree. Step 7 is not mandatory, it is the modification introduced by George to the initial algorithm (it has the purpose of further reducing the profile of a matrix, but that is another story). If you don't reverse the result array you end up with the original Cuthill-McKee algorithm.

Here is how the RCM algorithm should perform on the 8x8 input matrix presented at the start of this post:
- At first we must choose a starting node. Nodes initially labeled with 1, 4 and 7 are equally good choices as all of them have the degree of 1.
- Let us start with node 1. We'll add 1 in R and its neighbours in Q in the increasing order of their degree: R=[1]; Q=[5];
- We extract the first element from Q. As 5 isn't in R we add it. We also add all the neighbours of 5, that are not in R, in Q in the increasing order of their degree: R[1,5]; Q=[3];
- We extract the first element from Q. 3 is not in R so we add 3 to R and all the neighbours of 3, that are not in R, to Q in the increasing order of their degree: R=[1,5,3]; Q=[2];
- We extract the first element from Q. 2 is not in R so we add 2 to R and all the neighbours of 2, that are not in R, to Q in the increasing order of their degree: R=[1,5,3,2]; Q=[6,8];
- We extract the first element from Q. 6 is not in R so we add 6 to R and all the neighbours of 6, that are not in R, to Q in the increasing order of their degree: R=[1,5,3,2,6]; Q=[8,8];
- We extract the first element from Q. 8 is not in R so we add 8 to R and all the neighbours of 8, that are not in R, to Q in the increasing order of their degree: R=[1,5,3,2,6,8]; Q=[8];
- We extract the first element from Q. 8 is in R so we do nothing: R=[1,5,3,2,6,8]; Q=[];
- Q is empty so we check the size of R. If the size of R equals 8 then we are done, otherwise there are still unexplored nodes (the graph is not connected). As the size of R is 6 we must restart from Step 1.
- We must again choose a starting node. Nodes labeled 4 and 7 are equally good start solutions as they both have a degree of 1.
- We choose node 4.
We'll add 4 in R and its neighbours in Q in the increasing order of their degree: R=[1,5,3,2,6,8,4]; Q=[7];
- We extract the first element from Q. 7 is not in R so we add 7 to R and all the neighbours of 7, that are not in R, to Q in the increasing order of their degree: R=[1,5,3,2,6,8,4,7]; Q=[];
- Q is empty so we check the size of R. The size of R is 8 so all the nodes have been relabeled.
- We reverse R and obtain the solution R=[7,4,8,6,2,3,5,1] with the following interpretation:


If you look closely at the two graph images at the start of the post you can see how the node which had the initial label of 7 will be relabeled with 1, the node which had the initial label of 4 will be relabeled with 2, the node which had the initial label of 8 will be relabeled with 3, and so on.
.
The only thing left now is to build the adjacency matrix of the relabeled graph. You should obtain the 8x8
output matrix presented at the start of this tutorial.  

P.S. For a Delphi implementation (source code included) of the Cuthill-McKee and RCM algorithms, as well as a comparison of these methods with a more modern aproach based on a genetic algorithm and simulated annealing please see this post.

P.P.S. Help required / Collaboration opportunity

I have recently (February 2013) been contacted by Dr. Lubomir ONDRIS (lubomir.ondris(at)gmail.com) who is a scientist from Slovakia that has dedicated a lot of effort over the years into studying the bandwidth reduction problem and improving the available algorithms that attempt to solve it.

He created a personal hybrid algorithm that is quite competitive on the Everstine and other mesh collections (i.e., common benchmarks for testing the performance of bandwidth reduction routines) and he would need access to non-artificial, fairly big (>500 nodes) and complicated (inter-connected) meshes (matrices) in order to further test and improve his algorithm. Typically, such matrices are encountered when solving very large equation systems that arise in finite element analysis.

If you can help Dr. Ondris in any way with regard to his data requirements, please contact him directly via the e-mail address provided above. Here is a short description, in Dr. Ondris' own words, of what he requires:
"Again, what I need are parts of standard input data for finite element programs: Finite elements with their node numbers and nodes with their degrees of freedoms. And, of course, the bandwidth and profile achieved by the renumbering routine used."
Hope that you can help & thank you very much!