Methods for combining networks are a subject of active research. Many
different methods with different purposes have been proposed. The properties
and relationships of these methods are just beginning to be understood. Some
methods, such as boosting, are remedies for underfitting. Other methods,
such as bagging, are mainly remedies for overfitting or instability.
Bayesian learning naturally leads to model averaging (Hoeting et al., 1999).
A good general reference is the book edited by Sharkey (1999), especially
the article by Drucker (1999). Regarding the effects of bagging and weight
decay used together, see Taniguchi and Tresp (1997). 

Here is a list of terms used for various methods of combining models, mostly
taken from Christoph M. Friedrich's web page (see below): 

 o Adaboost 
 o ADDEMUP 
 o arcing: adaptive recombination of classifiers 
 o bagging: bootstrap aggregation 
 o bag-stacking: bagging plus stacking 
 o boosting 
 o cascading 
 o combination of classifiers 
 o committees of networks 
 o consensus theory 
 o cragging: cross aggregation (like k-fold cross validation) 
 o dagging: disjoint-sample aggregation 
 o dag-stacking: dagging plus stacking 
 o divide and conquer classifiers 
 o ensembles 
 o hagging: half-sample aggregation 
 o mixture of experts 
 o multiple classifier systems: 
 o multi-stage and multi-level classifiers 
 o OLC: optimal linear combination 
 o pandemonium of reflective agents 
 o sieving algorithms 
 o stacking: feeding outputs of several models (and possibly the the
   original inputs) into a second-level model 
 o voting 

URLs: 

 o Christoph M. Friedrich's web page, "Combinations of Classifiers and
   Regressors Bibliography and Guide to Internet Resources" at 
   http://www.tussy.uni-wh.de/~chris/ensemble/ensemble.html 

 o Tirthankar RayChaudhuri's web page on combining estimators at 
   http://www-comp.mpce.mq.edu.au/~tirthank/combest.html 

 o Robert E. Schapire's boosting page at 
   http://www.research.att.com/~schapire/boost.html 

 o http://www.boosting.org/ 

References: 

   Clemen, Robert T. (1989), "Combining forecasts: A review and annotated
   bibliography", International Journal of Forecasting, Vol 5, pp 559-584. 

   Drucker, H. (1999), "Boosting using neural networks," in Sharkey (1999),
   pp. 51-78. 

   Hoeting, J. A., Madigan, D., Raftery, A.E., and Volinsky, C.T. (1999)
   "Bayesian Model Averaging: A Tutorial (with discussion)," Statistical
   Science, 14:4, 382-417. Corrected version available at 
   http://www.stat.washington.edu/www/research/online/hoeting1999.pdf 

   Sharkey, A.J.C. (1999), Combining Artificial Neural Nets: Ensemble and
   Modular Multi-Net Systems, London: Springer. 

   Taniguchi, M., and Tresp, V. (1997), "Averaging regularized estimators,"
   Neural Computation, 9, 1163-1178. 

Some parts © 2009 Advameg, Inc.