A flux capacitor is a component associated with the starship's warp matrix.
After telling Dr. Bashir he was leaving Deep Space 9 to teach at the Academy, Miles O'Brien jokingly claimed it was because "somebody has to teach you officers the difference between a warp matrix flux capacitor and a self-sealing stembolt". (DS9: "What You Leave Behind")
When Lieutenant Barclay was offering hypothetical reasons for one of the anti-grav units' failure, Wesley Crusher suggested that Barclay check the "flow capacitor", which flustered Barclay. Barclay then visited one of his fantastical Holodeck programs and ranted to a simulated Counselor Troi about the encounter, "I knew about the flux capacitor, but I didn't need to hear about it from some 17-year-old kid." (TNG: "Hollow Pursuits")
See also
Background
The existence of a "flux capacitor" in the Trek universe is a light-hearted nod to the Back to the Future series of films, in which the device enables time travel. In the original film, Dr. Emmett Brown (Christopher Lloyd) envisions it after slipping off his toilet and hitting his head while hanging a clock.
Barclay's misuse of the phrase may be exclusively an in-joke by the writers or actor, but it may also be meant to undermine the character's intelligence.