A hostile witness was a witness called to testify against their position in a trial.
In 2365, while serving to defend Data against Bruce Maddox's request that the android be removed to the Daystrom Institute and disassembled for further analysis, Jean-Luc Picard called Maddox to the stand as a hostile witness. (TNG: "The Measure Of A Man")