Mysticism referred to a variety of different religious beliefs and practices, often associated with magic or some otherworldy supernatural forces.
In the case of the Vulcan people, it referred to certain practices such as their unique mental abilities, such as psychokinesis, telepathy, and their ability to transfer their own katra to other host bodies.
In 2285, Admiral Morrow told James T. Kirk that he didn't know if he believed in Vulcan mysticism. (Star Trek III: The Search for Spock)