Heresy was a term for beliefs contrary to established doctrine, also known as dogma. This dogma might be religious in nature. Some cultures viewed heresy as a crime. Someone espousing heresy was called a heretic.
In 2153, a group of Triannons seized control of Enterprise NX-01, hoping to eliminate a rival faction. Believing that the Delphic Expanse, or "Chosen Realm," was created in nine days, they considered the rival faction's belief that it was created in ten days to be heresy. Furthermore, their leader, Pri'Nam D'Jamat, believed that any questioning of his word by his followers could be considered heresy. (ENT: "Chosen Realm")
The Voth Ministry of Elders charged Professor Forra Gegen with the crime of heresy for teaching Distant Origin Theory, a scientific theory that challenged Voth religious doctrine. Despite the evidence standing before them, Professor Gegen was still found guilty. (VOY: "Distant Origin")
In 2373, Doctor Julian Bashir caused great commotion at a medical conference with a paper that stepped on some toes. He later explained to Jake Sisko that it was heresy to even consider the possibility that prion replication could be inhibited by quantum resonance effects. (DS9: "Nor the Battle to the Strong")
Miles O'Brien once confided that it was almost like heresy to dislike ship's counselors, and to feel that sometimes just got in the way. (DS9: "The Sound of Her Voice")
Edit