Recurrent neural networks (RNNs) are a class of powerful open system models in machine learning that are specially designed to learn sequential data. They have also enjoyed synergies with disciplines such as signal processing, optimization, systems and control theory, neuroscience and network science. The explosion of real-time data (physical or not) and the promising potential of using dynamical systems (physically realizable or not) for computation and to learn the data are opening up a wide range of foundational and practical problems. However, the dearth of rigorous analysis limits the usefulness of RNNs in addressing scientific questions. Therefore, a deep understanding of the working mechanism of RNNs and related models is pivotal to shed light on the properties of large and adaptive architectures, and to facilitate systematic design of the next generation of networks.