A topological anomaly was a type of invasive program, a paradoxical geometric form designed to overwhelm a computer's processing functions by spawning an infinite number of interlocking anomalous solutions.
In 2368, Lieutenant Commanders Data and Geordi La Forge designed such a program, designated topological anomaly 4747, as a weapon against the Borg. They planned to imprint the program onto the eyepiece of Third of Five, and from him it would spread to the rest of the Borg Collective. The program was expected to cause "total system failure" among the Borg after several hundred computational cycles.
However, Captain Picard eventually decided against using Third of Five, who had came to be known as "Hugh", to deploy the program. Hugh had began to reassert his individuality, and Picard believed it unethical to use a sentient being as an agent for mass destruction, even against the Borg. After Hugh elected to return to the Borg, Picard hoped that his individuality would have a similar disruptive effect on the Collective. (TNG: "I Borg")
Several months later, upon a new Borg incursion into the Federation, Vice Admiral Alynna Nechayev was angered and dismayed at Picard's decision to not release the program. She ordered him to act otherwise should another opportunity present itself. (TNG: "Descent")