ALOPEX

ALOPEX (an abbreviation of "algorithms of pattern extraction") is a correlation based machine learning algorithm first proposed by Tzanakou and Harth in 1974.

In machine learning, the goal is to train a system to minimize a cost function or (referring to ALOPEX) a response function. Many training algorithms, such as backpropagation, have an inherent susceptibility to getting "stuck" in local minima or maxima of the response function. ALOPEX uses a cross-correlation of differences and a stochastic process to overcome this in an attempt to reach the absolute minimum (or maximum) of the response function.

ALOPEX, in its simplest form is defined by an updating equation:

${\displaystyle \Delta \ W_{ij}(n)=\gamma \ \Delta \ W_{ij}(n-1)\Delta \ R(n)+r_{i}(n)}$

where:

• ${\displaystyle n\geq 0}$ is the iteration or time-step.
• ${\displaystyle \Delta \ W_{ij}(n)}$ is the difference between the current and previous value of system variable ${\displaystyle \ W_{ij}}$ at iteration ${\displaystyle n}$.
• ${\displaystyle \Delta \ R(n)}$ is the difference between the current and previous value of the response function ${\displaystyle \ R,}$ at iteration ${\displaystyle n}$.
• ${\displaystyle \gamma }$ is the learning rate parameter ${\displaystyle (\gamma \ <0}$ minimizes ${\displaystyle R,}$ and ${\displaystyle \gamma \ >0}$ maximizes ${\displaystyle R\ )}$
• ${\displaystyle r_{i}(n)\sim \ N(0,\sigma \ ^{2})}$

Essentially, ALOPEX changes each system variable ${\displaystyle W_{ij}(n)}$ based on a product of: the previous change in the variable ${\displaystyle \Delta }$${\displaystyle W_{ij}(n-1)}$, the resulting change in the cost function ${\displaystyle \Delta }$${\displaystyle R(n)}$, and the learning rate parameter ${\displaystyle \gamma }$. Further, to find the absolute minimum (or maximum), the stochastic process ${\displaystyle r_{ij}(n)}$ (Gaussian or other) is added to stochastically "push" the algorithm out of any local minima.

• Harth, E., & Tzanakou, E. (1974) Alopex: A stochastic method for determining visual receptive fields. Vision Research, 14:1475-1482. Abstract from ScienceDirect