Centorrino, Veronica (2024) Contracting Dynamics for Biologically Plausible Neural Networks and Optimization. [Tesi di dottorato]

Documento PDF centorrino_veronica_36.pdf Download (6MB)

Abstract

Our brain is perhaps one of the most striking examples of complex systems: about $10^{10}$ neurons, interconnected by approximately $10^{15}$ \emph{recurrent} synaptic connections, capable of adapting and learning through \emph{local synaptic rules}, continuously solving \emph{optimization problems} like sparse representation. While artificial neural networks (ANN) were initially inspired by natural networks, nowadays they have significantly diverged from biological realism, driven by performance criteria. As a result, ANNs still exhibit errors and biases that are absent in natural networks and that cannot be explained and quantified a priori. This naturally raises important questions: What if we could design ANNs with today’s performance, but that more closely mimic the way natural networks work? How can we obtain biologically plausible models and how can we guarantee their stability and robustness? What optimization problems do natural networks solve, and how can biologically plausible neural networks mimic these processes? Can we create a normative framework translating optimization problems into ANNs that are guaranteed to converge to equilibria representing optimal solutions of the initial optimization problems? Motivated by these exciting open challenges, in this thesis we aim at laying the theoretical groundwork for the modeling and understanding of neural networks that align more closely with biological principles. We propose a normative framework that translates complex tasks, described as optimization problems, into biologically plausible neural networks that are guaranteed to converge to equilibria corresponding to the optimal solutions. Our models incorporate core principles of natural networks: the use of continuous-time dynamical systems for both neural and synaptic changes, recurrent connections, positivity of the system, and local learning rules. Specifically, for the neural dynamics we focus on two widely used recurrent neural network (RNN) models -- the Hopfield neural network (HNN) and the firing rate neural network (FNN) -- and model synaptic weight dynamics using continuous-time Hebbian learning rules. To ensure stability and robustness of our models, we leverage \emph{contraction theory}, a robust computationally-friendly stability tool from control theory. This approach offers a significant advantage: with a single condition, it guarantees global exponential convergence, along with a number of highly ordered transient and asymptotic behaviors of contracting dynamics, which are advantageous for our objectives. We begin our analysis by developing essential theoretical tools to analyze relevant neural dynamics, filling a gap in the literature by establishing conditions for strong and weak Euclidean contractivity of RNNs with locally Lipschitz activation functions. Remarkably, our lower bound on the contraction rate is log-optimal for almost all symmetric weight matrices, making our results sharp -- they are the best achievable within this framework. Additionally, we establish new algebraic results on matrix polytopes and symmetric matrix products, advancing both neural network stability and broader applications in matrix theory and optimization. Building on these theoretical results, we propose a top-down normative framework for designing biologically plausible networks that solve sparse reconstructions and other optimization problems. This framework is based upon the theory of proximal operators for composite optimization and leads to continuous-time firing rate neural networks -- the \emph{firing rate competitive network} -- that are therefore interpretable. We analyze the behavior of these dynamics, establishing a direct link between the network equilibria and the optimal solutions of sparse reconstruction problems. Under a standard assumption, we prove that the dynamics converge linear-exponentially (and thus globally) to the equilibrium. Importantly, we show that the positive variant of the firing rate competitive network preserves non-negativity in its state variables, aligning with biological principles and further underscoring the plausibility of the model. In the second part, we explore the interaction between neural and synaptic dynamics by embedding Hebbian learning rules into the continuous-time RNN models of the first part. This results in the \emph{coupled neural-synaptic networks:} RNNs with dynamic synaptic connections that closely mirror biological processes. We propose and analyze these systems, combining HNNs and FNNs with Hebbian learning rules. For these dynamics, we propose a low-dimensional formulation that captures synaptic sparsity of neural circuits. We establish sufficient conditions for the contractivity of each model by leveraging non-Euclidean contraction arguments. Additionally, we demonstrate biologically plausible forward invariance results and show that under suitable conditions, the models satisfy Dale’s Principle -- an empirical principle referring to the fact that a neuron has either only excitatory or only inhibitory synapses -- further enhancing biological plausibility. In the final part of the thesis, we explore the potential of a contractivity-based approach for optimization. We begin by translating canonical static optimization problems into continuous-time dynamical systems, establishing conditions for strong infinitesimal contractivity. For both static and time-varying optimization problems, we derive contractivity conditions and, in certain cases, demonstrate improved convergence rates. Our work includes two key results on equilibrium tracking in parameter-varying contracting dynamics, addressing both known and unknown rates of parameter change. Additionally, we extend our analysis to convex optimization problems with unique minimizers. We show that these problems lead to dynamics that are globally-weakly contracting in the state space and only locally-strongly contracting. For these dynamics, we present a detailed convergence analysis, showing that convergence is linear-exponential. This means that the distance between each solution and the equilibrium is upper-bounded by a function that first decreases linearly and then exponentially. We also provide input-to-state stability conditions for these dynamics, further strengthening the robustness and applicability of the approach. We conclude by highlighting potential future directions and with an appendix with novel complementary results on the Euclidean contractivity of FNNs with dissipation.

Downloads

Actions (login required)

Modifica documento