Protective custody was the detention of an individual for their own safety's sake, rather than punishment for a crime.
In 2270, Captain James T. Kirk "arrested" Commander Bem in this manner, saying he wished to keep him from getting into further trouble. (TAS: "Bem")
In 2287, in a video recording, Caithlin Dar described herself and the rest of Sybok's converts as being in the Galactic Army of Light's protective custody. (Star Trek V: The Final Frontier)
In 2373, Odo placed Arissa in protective custody in his quarters to keep her from being murdered. His choice of venue led Jadzia Dax to assume (rightly) that something happened between the couple, though Kira Nerys was skeptical, saying he was only being thorough. (DS9: "A Simple Investigation")
In 2381, Nova One began pursuing ex-Starfleet personnel, prompting Starfleet to assign ships to search for Seven of Nine, Beverly Crusher, Thomas Riker, and Nick Locarno. The USS Cerritos was assigned to search for Locarno, who had moved to New Axton to work as a pilot for hire following his expulsion from Starfleet Academy. When a team led by Captain Carol Freeman found Locarno's hideout, however, Locarno was nowhere to be found, and Lieutenant jg Sam Rutherford discovered schematics for the mysterious ship, proving that Locarno himself was behind the thefts. (LD: "The Inner Fight")