A behavioral disorder was a psychological disorder that affected one's behavior.
Jack, Lauren, and Patrick all suffered from various behavioral disorders as a result of their DNA resequencing. Doctor Julian Bashir did not believe Sarina Douglas suffered from the same type of disorders as theirs, saying Douglas was trapped in her own mind. (DS9: "Chrysalis")