This section gives a quick introduction to the mathematical foundations of deep learning:
Activation Functions
Optimization & backpropagation
Mathematical Foundations
Resources & references