The work presented in this thesis was initially motivated by the discrepancy between the impressive performances of modern neural networks and the lack of applications to scientific problems for which data abounds. Focusing on evolution problems which are classically modelled through ordinary or partial differential equations~(O/PDEs) naturally brought us to consider the more general problem of representing and learning such equations from raw data with neural networks. This was the inception of the first part of our work. The point of view considered in this first part has a natural counterpart: what about the dynamics induced by the trajectories of the NN's weights during training or by the trajectories of data points within them during inference? Can they be usefully modelled? This question was the core of the second part of our work and, while theoretical tools other than O/PDEs happened to be useful in our analysis, our reasoning and intuition were fundamentally driven by considerations stemming from a dynamical viewpoint.