A mathematician was one who studied mathematics.
Some species were better in mathematics than others. The Qomar in particular were a species of mathematicians, to the point that the number of digits of pi one could calculate was of social interest on their homeworld. (VOY: "Virtuoso") Taresians also claimed to have a gift for math. (VOY: "Favorite Son")
Jean-Luc Picard enjoyed trying to prove Fermat's last theorem, because to him "it puts things in perspective," clarifying that, "in our arrogance, we feel we are so advanced, and yet we cannot unravel a simple knot tied by a part-time French mathematician, working alone, without a computer." (TNG: "The Royale")