Protective custody was the detention of an individual for their own safety's sake, rather than punishment for a crime.
In 2270, James T. Kirk "arrested" Commander Bem in this manner, saying he wished to keep him from getting into further trouble. (TAS: "Bem")
In 2287, in a video recording, Caithlin Dar described herself and the rest of Sybok's converts as being in the Galactic Army of Light's protective custody. (Star Trek V: The Final Frontier)
In 2373, Odo placed Arissa in protective custody in his quarters to keep her from being murdered. His choice of venue led Jadzia Dax to assume (rightly) that something happened between the couple, though Kira Nerys was skeptical, saying he was only being thorough. (DS9: "A Simple Investigation")