A jack-in-a-box, as Chief Miles O'Brien phrased it, was "a children's toy, on Earth. You wind it up and a figure jumps out and goes 'Boo!'."
In holo-programming, a jack-in-the-box was a hidden subroutine that a programmer could add to an interactive holosuite program, which was designed to suddenly spring to life, just as the children's toy of the same name, to surprise the user and keep the program from getting boring. However, unlike the children's toy, which merely says, "Boo", a holosuite jack-in-the-box can cause significant chaos to the very program which it is a part of. The results of a jack-in-the-box can lead to the termination or corruption of the program itself.
Although not self-aware, the characters in a jack-in-the-box can be made immune from re-programming and from "freeze program" commands. Therefore, once activated, the subroutine runs in real-time, making it possible for events to transpire without any real-world (non-holographic) character being present. Jack-in-the-box characters can be made "period-specific", meaning that only items and methods available in the program's universe are usable against the character (thus, a phaser would have been useless against a period-specific 1960s gangster).
Often, the designer of a jack-in-the-box will include an "out" (that is, a way to beat the jack-in-the-box). This "out" must be accomplished within the boundaries of the program's universe. The opportunity to reach an "out" may also expire, since the characters in a jack-in-the-box are "existing" in real-time, and potentially changing the components of their own existence.
Julian Bashir's friend Felix created a jack-in-the-box as part of his Vic Fontaine Las Vegas lounge program. When he and the crew of Deep Space 9 became too comfortable with Vic, taking him for granted, a period-specific jack-in-the-box was suddenly activated to "add some spice to the story."
Frankie Eyes, a mobster with a deep-seated grudge against Vic, as well as muscleman Tony Cicci sprang into "existence," and took over Vic's lounge. Backed by the wealthy, yet equally corrupt, Carl Zeemo, they threatened Vic's very existence in the program, and were only defeated by the combined effort of the space station's senior staff in 2375. (DS9: "Badda-Bing, Badda-Bang")
In 2371, Seska wrote a deadly jack-in-the-box into Tuvok's tactical training scenario Insurrection Alpha that, when Tuvok next opened the narrative parameters file, would trap Tuvok and Tom Paris in the holodeck and turn off the safety protocols. Then the Seska character hunted them down. Captain Janeway managed to rewrite the story to get Tuvok and Paris out, but Seska's additions adapted. The out for this jack-in-the-box was to kill the Seska character, which Tuvok did by rigging a phaser malfunction. (VOY: "Worst Case Scenario")
This is roughly analogous to the Easter egg concept in modern computer programming.