Search the FAQ Archives

3 - A - B - C - D - E - F - G - H - I - J - K - L - M
N - O - P - Q - R - S - T - U - V - W - X - Y - Z
faqs.org - Internet FAQ Archives

comp.ai.neural-nets FAQ, Part 3 of 7: Generalization
Section - How to combine networks?

( Part1 - Part2 - Part3 - Part4 - Part5 - Part6 - Part7 - Single Page )
[ Usenet FAQs | Web FAQs | Documents | RFC Index | Neighborhoods ]


Top Document: comp.ai.neural-nets FAQ, Part 3 of 7: Generalization
Previous Document: What is Bayesian Learning?
Next Document: How many hidden layers should I use?
See reader questions & answers on this topic! - Help others by sharing your knowledge

Methods for combining networks are a subject of active research. Many
different methods with different purposes have been proposed. The properties
and relationships of these methods are just beginning to be understood. Some
methods, such as boosting, are remedies for underfitting. Other methods,
such as bagging, are mainly remedies for overfitting or instability.
Bayesian learning naturally leads to model averaging (Hoeting et al., 1999).
A good general reference is the book edited by Sharkey (1999), especially
the article by Drucker (1999). Regarding the effects of bagging and weight
decay used together, see Taniguchi and Tresp (1997). 

Here is a list of terms used for various methods of combining models, mostly
taken from Christoph M. Friedrich's web page (see below): 

 o Adaboost 
 o ADDEMUP 
 o arcing: adaptive recombination of classifiers 
 o bagging: bootstrap aggregation 
 o bag-stacking: bagging plus stacking 
 o boosting 
 o cascading 
 o combination of classifiers 
 o committees of networks 
 o consensus theory 
 o cragging: cross aggregation (like k-fold cross validation) 
 o dagging: disjoint-sample aggregation 
 o dag-stacking: dagging plus stacking 
 o divide and conquer classifiers 
 o ensembles 
 o hagging: half-sample aggregation 
 o mixture of experts 
 o multiple classifier systems: 
 o multi-stage and multi-level classifiers 
 o OLC: optimal linear combination 
 o pandemonium of reflective agents 
 o sieving algorithms 
 o stacking: feeding outputs of several models (and possibly the the
   original inputs) into a second-level model 
 o voting 

URLs: 

 o Christoph M. Friedrich's web page, "Combinations of Classifiers and
   Regressors Bibliography and Guide to Internet Resources" at 
   http://www.tussy.uni-wh.de/~chris/ensemble/ensemble.html 

 o Tirthankar RayChaudhuri's web page on combining estimators at 
   http://www-comp.mpce.mq.edu.au/~tirthank/combest.html 

 o Robert E. Schapire's boosting page at 
   http://www.research.att.com/~schapire/boost.html 

 o http://www.boosting.org/ 

References: 

   Clemen, Robert T. (1989), "Combining forecasts: A review and annotated
   bibliography", International Journal of Forecasting, Vol 5, pp 559-584. 

   Drucker, H. (1999), "Boosting using neural networks," in Sharkey (1999),
   pp. 51-78. 

   Hoeting, J. A., Madigan, D., Raftery, A.E., and Volinsky, C.T. (1999)
   "Bayesian Model Averaging: A Tutorial (with discussion)," Statistical
   Science, 14:4, 382-417. Corrected version available at 
   http://www.stat.washington.edu/www/research/online/hoeting1999.pdf 

   Sharkey, A.J.C. (1999), Combining Artificial Neural Nets: Ensemble and
   Modular Multi-Net Systems, London: Springer. 

   Taniguchi, M., and Tresp, V. (1997), "Averaging regularized estimators,"
   Neural Computation, 9, 1163-1178. 

User Contributions:

Comment about this article, ask questions, or add new information about this topic:

CAPTCHA




Top Document: comp.ai.neural-nets FAQ, Part 3 of 7: Generalization
Previous Document: What is Bayesian Learning?
Next Document: How many hidden layers should I use?

Part1 - Part2 - Part3 - Part4 - Part5 - Part6 - Part7 - Single Page

[ Usenet FAQs | Web FAQs | Documents | RFC Index ]

Send corrections/additions to the FAQ Maintainer:
saswss@unx.sas.com (Warren Sarle)





Last Update March 27 2014 @ 02:11 PM