Penance was an act of voluntary self-punishment done out of repentance for sins one committed that was also comparable to confession.
When Q recalled in 2372, Q's being made mortal following his banishment from the Q Continuum for his crimes in 2366, Q explained that his "penance has ended" and that he was "a born again Q." (VOY: "Death Wish")
In 2374, after performing a miracle for Benjamin Sisko after he asked for their help in dealing with the Dominion fleet that was crossing through the Bajoran wormhole, the Prophets warned him that there would be a penance, that he was of Bajor but would find no rest there. (DS9: "Sacrifice of Angels") This later proved true when, while falling to his death, Sisko was transported to the wormhole, or Celestial Temple, where he was to remain with the Prophets, though he told Kasidy Yates-Sisko that he would one day be able to return. (DS9: "What You Leave Behind")