This work systematically studies number understanding in language models and proposes effective methods to improve numerical reasoning capabilities.