A vortex was a rift or tear within subspace anomalies. Vortexes compresses subspace and any other matter with which it comes into contact. Matter compressed by the vortex effectively miniaturizes or "shrinks" that matter, although the miniaturization can be reversed so long as the matter leaves the anomaly along the same path by which it entered.
Crew members from Deep Space 9 investigated one such vortex aboard the USS Rubicon in 2374. The ship and its crew were miniaturized but were forced to flee the anomaly using a different path, and so they remained miniature after leaving the anomaly. They were later able to grow back to their normal size by re-entering the anomaly and following their original trajectory back out. (DS9: "One Little Ship")