TY - JOUR
T1 - Entropies of Overcomplete Kernel Dictionaries
AU - Honeine, Paul
JF - Bulletin of Mathematical Sciences and Applications
VL - 16
SP - 1
EP - 19
SN - 2278-9634
PY - 2016
PB - SciPress Ltd
DO - 10.18052/www.scipress.com/BMSA.16.1
UR - https://www.scipress.com/BMSA.16.1
KW - Dictionary Learning
KW - Generalized Rényi Entropy
KW - Gram Matrix
KW - Kernel-Based Methods
KW - Machine Learning
KW - Pattern Recognition
KW - Shannon Entropy
KW - Sparse Approximation
KW - Tsallis Entropy
AB - In signal analysis and synthesis, linear approximation theory considers a linear decomposition of any given signal in a set of atoms, collected into a so-called dictionary. Relevant sparse representations are obtained by relaxing the orthogonality condition of the atoms, yielding overcomplete dictionaries with an extended number of atoms. More generally than the linear decomposition, overcomplete kernel dictionaries provide an elegant nonlinear extension by defining the atoms through a mapping kernel function (e.g., the gaussian kernel). Models based on such kernel dictionaries are used in neural networks, gaussian processes and online learning with kernels. The quality of an overcomplete dictionary is evaluated with a diversity measure the distance, the approximation, the coherence and the Babel measures. In this paper, we develop a framework to examine overcomplete kernel dictionaries with the entropy from information theory. Indeed, a higher value of the entropy is associated to a further uniform spread of the atoms over the space. For each of the aforementioned diversity measures, we derive lower bounds on the entropy. Several definitions of the entropy are examined, wth an extensive analysis in both the input space and the mapped feature space.
ER -