Fabiani, Gianluca (2024) Machine learning-based modelling and numerical analysis of the emergent dynamics of complex systems. [Tesi di dottorato]

Documento PDF Fabiani_Gianluca_36.pdf Download (12MB)

Abstract

The aim of this Doctoral Thesis is the development of new methodologies at the intersection of machine learning (ML) and numerical analysis to address key challenges in modeling, understanding, and forecasting the emergent dynamics of complex systems. In particular, by integrating these two fields, we overcome key limitations of traditional and deep learning approaches linked to the “curse of dimensionality” in their training, even for simple problems. The core focus of the Thesis lies in solving both forward and inverse problems, while also providing new theoretical and practical contributions to the mathematical foundations of ML. A critical aspect of this research focuses on the advancement of ML methodologies to study complex systems, thus dealing with the so-called “curse of dimensionality”. In particular, we focus on Random Projection Neural Networks (RPNNs), which relax the “curse of dimensionality” and the challenges in finding global optima for non-convex training process of fully-trained Feedforward Neural Networks (FNNs) for dynamical systems. Throughout this Thesis, we have theoretically analyzed the RPNNs for function approximation, demonstrating their exponential convergence rate for smooth functions. The Thesis has two branches: the theoretical advances and their applications for complex systems. Of course, these two are in dialectic relation. Applications drive the development of new methods, and new methods provide tools for understanding better the behavior of such systems, and so forth and so on. From the theoretical point of view, we investigated and developed various methods for sampling the internal weights and biases of RPNNs, employing both function-agnostic and function-informed approaches while developing a geometric-based sampling algorithm for high-dimensional problems. Furthermore, we have introduced Random Projection-based Operator Networks (RandONets), which extend the use of random projections for Deep Operator Networks (DeepONets). We theoretically prove for RandONets an extension of the Chen and Chen's theorem of universal approximation for operators and empirically demonstrate that randomized and a priori fixed embeddings of both branch and trunk hidden layers can outperform traditional fully-trained DeepONets counterparts by several orders of magnitude in both computational time and numerical accuracy. The above enhance our arsenal to solve both the forward and inverse problem in complex systems. The forward problem focuses on the numerical solution of differential equations (DEs) for complex systems and their associated challenging issues, such as those involving stiff systems and multiscale simulations, as well as analyzing system stability through bifurcation analysis, deeply connected with the analysis of multi-stability phenomena. We rigorously evaluate ML methods, particularly Physics-Informed RPNNs (PIRPNNs), against established numerical techniques. Our results show that parsimoniously designed PIRPNNs can match or outperform traditional numerical methods and deep learning approaches in accuracy and efficiency, especially in handling stiff systems and sharp gradients. In these forward problems, by exploiting analytical derivatives of the network, solving with RPNNs reduces to finding least-squares solutions of nonlinear algebraic equations. To enhance training efficiency, we propose replacing computationally expensive gradient-based algorithms with Newton's method, combined with specialized techniques such as Moore-Penrose pseudo-inverses based on singular value decomposition (SVD) or Complete Orthogonal Decomposition (COD). We stress, that the developing of such novel algorithms and training strategies, employing for example random projections, aim to enhance the reliability and efficiency of ML for complex systems modeling. For illustration purposes, we have applied these PIRPNNs to stationary Partial Differential Equations (PDEs), including the 1D and 2D Liouville-Bratu-Gelfand and viscous Burgers' equations with mixed boundary conditions. In addition, we coupled this solution with arc-length continuation in the RPNN weight space to construct accurate bifurcation diagrams. In these experiments, we outperformed established techniques, such as finite difference (FD) and finite element methods (FEM), for the first time. Our research also extended to solving Ordinary Differential Equations (ODEs) and Differential Algebraic Equations (DAEs), including various stiff benchmark problems and stiff PDEs, including van der Pol ODEs, Robertson DAEs, Belousov-Zhabotinsky ODEs and Allen-Cahn phase field PDE discretized with FD. Notably, we surpassed a professional Matlab solver, ode15s, which is based on the backward differentiation formula (BDF) of variable order and variable step sizes. On the other hand, the solution of the inverse problem (reconstruction of differential equations from data), is crucial for the parameter identification, system reconstruction, and discovering hidden dynamics in complex systems. Inverse problems often pose additional challenges, such as ill-posedness and sensitivity to noise, requiring robust and efficient approaches. The primary goal here is to infer underlying properties of the differential equations from observed data — specifically, identifying the macroscopic governing laws of emergent behaviors. This involves constructing ML-assisted surrogate models in the form of DEs, possibly identifying the presence of stochastic terms (SDEs), partial derivatives (PDEs), and/or integral features (IPDEs). To achieve this, a significant challenge lies in identifying coarse-grained observables that accurately describe the emergent behavior. This step is critical for capturing the key dynamics of the system while reducing the complexity and high-dimensionality of the underlying data. We employ manifold learning techniques to systematically extract relevant macroscopic observables from high-dimensional data, ensuring that these observables are both significant and representative of the system's emergent behavior. Based on the selected significant coarse-scale observables, we learn the right-hand-side (RHS) of the effective DEs using two ML schemes, namely FNNs and RPNNs. To handle challenges related to dominant spatial features, we combine these neural architectures with fixed convolutional layers based on finite difference stencils, which can provide new feature candidates. We further refine the selection of these features through parsimonious Diffusion Maps (DMaps) and Automatic Relevance Determination (ARD) with Gaussian Processes (GPs). We contrast the advantages and limitations of different ML surrogate models, emphasizing the importance of developing task-oriented models. Specifically, we highlight the trade-off between the computational endeavor in learning infinite-dimensional PDEs operators and the simplicity of low-dimensional targeted surrogates, in the form of SDEs, particularly for tasks such as identifying tipping points (bifurcation points in dynamical systems) and performing rare event analysis in their neighborhood. In particular, for this task, we utilize both brute force Monte Carlo simulations and statistical mechanics-based methods. For the task-oriented SDE surrogate, we employ two coupled FNNs inspired by the Euler-Maruyama scheme to simultaneously learn the drift and diffusivity functions of an underlying Wiener process, based on the assumption that the data follow such a process. Notably, these low-dimensional surrogates accelerate rare event analysis by three orders of magnitude compared to full-scale microscopic simulations. This thesis presents three illustrative case studies for inverse problems: (i) the spatial propagation of action potentials in unmyelinated neurons, modeled by the FitzHugh-Nagumo PDEs, reconstructed from a mesoscopic lattice Boltzmann description; (ii) an event-driven agent-based model capturing mimesis-driven behavior; and (iii) the spread of an epidemic on an Erdős-Rényi network. Ultimately, this thesis contributes to the advancement of both ML and numerical analysis by providing novel insights into their synergistic potential. The proposed methodologies offer promising avenues for tackling complex systems challenges across a wide range of applications, such as neuroscience, finance, and epidemiology.

Downloads

Actions (login required)

Modifica documento