It's a language, yes, but even languages are analogies /
   metaphors.

   it's nice that mathematics corresponds well to the skeletal form
   of the physical Universe so far, although we've hit snags where
   the mathematics analogy breaks down and mathematics ceases to
   work as a useful descriptor for observation. But it's a very
   practical language for a good portion of it.

   Cognitively, we have two sets of number systems in the brain.
   One is an instant form; typically happens on the left side of
   the brain if I'm not mistaken - numosity I think it's called; we
   recognize numbers up to 3 (sometimes 4) instantly. But anything
   beyond that, the semantic systems of the brain takes over.
   Roughly same areas as language, analogy, metaphor, poetry,
   grammar, etc.