|
Top Document: comp.ai.neural-nets FAQ, Part 2 of 7: Learning Previous Document: Why use activation functions? Next Document: What is a softmax activation function? See reader questions & answers on this topic! - Help others by sharing your knowledge The formula for the logistic activation function is often written as: netoutput = 1 / (1+exp(-netinput)); But this formula can produce floating-point overflow in the exponential function if you program it in this simple form. To avoid overflow, you can do this: if (netinput < -45) netoutput = 0; else if (netinput > 45) netoutput = 1; else netoutput = 1 / (1+exp(-netinput)); The constant 45 will work for double precision on all machines that I know of, but there may be some bizarre machines where it will require some adjustment. Other activation functions can be handled similarly. User Contributions:Comment about this article, ask questions, or add new information about this topic:Top Document: comp.ai.neural-nets FAQ, Part 2 of 7: Learning Previous Document: Why use activation functions? Next Document: What is a softmax activation function? Part1 - Part2 - Part3 - Part4 - Part5 - Part6 - Part7 - Single Page [ Usenet FAQs | Web FAQs | Documents | RFC Index ] Send corrections/additions to the FAQ Maintainer: saswss@unx.sas.com (Warren Sarle)
Last Update March 27 2014 @ 02:11 PM
|
comp.ai.neural-nets FAQ, Part 2 of 7: Learning
Section - How to avoid overflow in the logistic function?
Search the FAQ Archives
N - O - P - Q - R - S - T - U - V - W - X - Y - Z
comp.ai.neural-nets FAQ, Part 2 of 7: Learning
Section - How to avoid overflow in the logistic function?
( Part1 - Part2 - Part3 - Part4 - Part5 - Part6 - Part7 - Single Page
)
[ Usenet FAQs | Web FAQs | Documents | RFC Index | Forum archive ]
PDP++ is a neural-network simulation system written in C++, developed as an advanced version of the original PDP software from McClelland and Rumelhart's "Explorations in Parallel Distributed Processing Handbook" (1987). The software is designed for both novice users and researchers, providing flexibility and power in cognitive neuroscience studies. Featured in Randall C. O'Reilly and Yuko Munakata's "Computational Explorations in Cognitive Neuroscience" (2000), PDP++ supports a wide range of algorithms. These include feedforward and recurrent error backpropagation, with continuous and real-time models such as Almeida-Pineda. It also incorporates constraint satisfaction algorithms like Boltzmann Machines, Hopfield networks, and mean-field networks, as well as self-organizing learning algorithms, including Self-organizing Maps (SOM) and Hebbian learning. Additionally, it supports mixtures-of-experts models and the Leabra algorithm, which combines error-driven and Hebbian learning with k-Winners-Take-All inhibitory competition. PDP++ is a comprehensive tool for exploring neural network models in cognitive neuroscience.