"Deep Networks are Reproducing Kernel Chains"

Deep networks reimagined through kernel composition, eliminating bottleneck layers while preserving mathematical properties.

This paper introduces a new mathematical framework called chain RKBS (Reproducing Kernel Banach Spaces) that preserves key properties of shallow neural networks when building deep networks through kernel composition instead of function composition.

-----

https://arxiv.org/abs/2501.03697

🤔 Original Problem:

Deep neural networks lack a proper mathematical function space framework that maintains desirable properties like reproducing kernels and sparsity through network depth.

-----

🔧 Solution in this Paper:

→ The paper extends Reproducing Kernel Banach Spaces (RKBS) to chain RKBS (cRKBS), composing kernels rather than functions

→ This new framework preserves RKBS properties through network depth while avoiding extra bottleneck layers

→ Neural cRKBS, a special subclass, directly represents neural networks through kernel chaining

→ The approach guarantees sparse solutions requiring no more than N neurons per layer for N data points

→ Weight-sharing capabilities emerge naturally through the kernel composition process

-----

💡 Key Insights:

→ Deep networks are not compositions of shallow networks due to extra bottleneck layers

→ Kernel composition matches hidden layers directly without bottlenecks

→ The framework provides a natural infinite-width limit for deep networks

→ cRKBS maintains mathematical rigor while being practically implementable

-----

📊 Results:

→ Proves that any deep neural network is a neural cRKBS function

→ Shows that any neural cRKBS function over finite data corresponds to a deep network

→ Achieves sparse solutions with at most N(N+1)(L+1) total parameters for L layers