The unreasonable effectiveness of neural network approximation

Results concerning the approximation rates of neural networks are of particular interest to engineers. The results reported in the literature have “slow approximation rates” O(1/√m), where m is the number of parameters in the neural network. However, many empirical studies report that neural network approximation is quite effective in practice. We give an explanation of this unreasonable effectiveness by proving the existence of approximation schemes that converge at a rate of the order of 1/m² by using methods from number theory