Introduction to Soft Computing

(PART 1)

Neural Networks-I

Sub-Topics
Neuron, Nerve structure and synapse, Artificial Neuron and its model, activation functions, Neural network architecture: single layer and multilayer feed forward networks, recurrent networks. Various learning techniques, perception and convergence rule, Auto-associative and hetero-associative memory.

Introduction and Architecture

The human brain is composed of countless neural cells, known as neurons, that process and transmit information. Each neuron functions like a simple processor, and the massive interactions between these cells enable the brain’s remarkable abilities.

Structure of a Neuron

1. Dendrites: These are branching fibers that extend from the neuron’s cell body (soma) and are responsible for receiving signals from other neurons.
2. Soma (Cell Body): The soma houses the nucleus and other essential structures that support chemical processing and neurotransmitter production.
3. Axon: This singular fiber carries information away from the soma to synaptic sites on other neurons, muscles, or glands.
4. Axon Hillock: This region acts as the summation point for incoming signals. The collective influence of signals received by the neuron determines whether an action potential will be generated and propagated along the axon.
5. Myelin Sheath: Composed of fat-containing cells, the myelin sheath insulates the axon, significantly increasing the speed of signal transmission. Gaps, known as Nodes of Ranvier, exist between myelin segments, allowing electrical signals to jump from one gap to the next.
6. Synapse: This is the connection point between two neurons or between a neuron and a muscle or gland. Here, electrochemical communication occurs, facilitating the transmission of signals.
7. Terminal Buttons: Located at the end of the axon, terminal buttons release neurotransmitters into the synapse, enabling communication with other neurons.

McCulloch-Pitts introduced a simplified model of this real neurons.

• The McCulloch-Pitts Neuron: A Simplified Model of Neural Function

The McCulloch-Pitts neuron is a foundational model in artificial intelligence and computational neuroscience, representing a simplified version of biological neurons. This model is known as a Threshold Logic Unit and serves as a basis for understanding how neurons process information.

Components of the McCulloch-Pitts Neuron

1. Synapses: A set of connections that receive activation signals from other neurons. These connections play a crucial role in integrating inputs.
2. Processing Unit: This unit sums the incoming signals and applies a non-linear activation function, known as the transfer function or threshold function.
3. Output Line: The processed result is transmitted to other neurons through the output line. When the accumulated signals reach a certain threshold, the neuron “fires” or discharges its output, after which it can begin to accumulate signals again.

Functions of the McCulloch-Pitts Neuron

The relationship between input and output in the McCulloch-Pitts neuron can be expressed as:

• Function: y=f(x) describes the mapping from input x to output y.
• Threshold or Sign Function (x): This function determines the output based on whether the input exceeds a defined threshold.
• Sigmoid Function: A smoothed, differentiable version of the threshold function, which allows for more gradual transitions and is widely used in various neural network architectures.

McCulloch-Pitts (M-P) Neuron Equation:

A Binary Model of Neural Activity

The McCulloch-Pitts (M-P) neuron model provides a simplified mathematical representation of a biological neuron. This model utilizes a binary approach, where the neuron receives inputs, processes them, and produces an output. Here are the key components of the M-P neuron model:

Components of the McCulloch-Pitts Neuron Model

1. Inputs: The M-P neuron accepts multiple binary inputs, denoted as x₁,x₂ ,…,x_n . Each input can either be 0 (inactive) or 1 (active).
2. Weights: Each input x​_i has an associated weight w_i, which indicates the strength of that input. These weights are typically binary as well, often set to 1 or 0.
3. Threshold: The neuron is defined by a threshold value θ. This threshold determines whether the neuron will activate based on the sum of its weighted inputs.
4. Output: The output y of the neuron is also binary:
   • If the sum of the weighted inputs exceeds the threshold, the output y is 1, indicating that the neuron “fires.”
   • If the sum does not exceed the threshold, the output y is 0, meaning the neuron does not fire.

Equation Representation

The M-P neuron can be mathematically expressed as follows:

Where:

θ is the threshold value.

y is the output of the neuron.

w_i is the weight of the input x_i​.

Activation Functions

Activation functions play a crucial role in neural networks by determining how the signal output is transformed. Choosing the right activation function is essential for the performance of your model, as it can significantly impact the network’s ability to learn and generalize. Here are some of the most common activation functions:

1. Linear Function: Simple and straightforward, linear activation functions allow for direct proportionality between input and output. However, they can limit the network’s ability to model complex relationships.
2. Threshold Function: This binary activation function outputs a fixed value based on whether the input surpasses a certain threshold, making it useful for classification tasks.
3. Piecewise Linear Function: Combining the benefits of linearity and non-linearity, piecewise linear functions can better capture varying relationships in data.
4. Sinusoidal (S-Shaped) Function: This function introduces non-linearity and is useful in scenarios where periodic patterns exist in the data.
5. Tangent Hyperbolic Function (Tanh): A popular choice, Tanh outputs values between -1 and 1, which helps to center the data and can improve convergence in training.

Choosing the appropriate activation function depends on the specific problem you are trying to solve with your neural network.