A cascade failure occurs when a "learning" program is corrupted, either by repeated application to a task it is ill suited to perform or by external influence such as damage to the hardware used to implement it.
In a neural net system, in particular a positronic brain, in which most or all sub-programs interact, this can lead to corruption of surrounding algorithms and routines as they attempt to use data generated by the faulty application to perform a task of their own. This may compound the damage to the original sub-program, as neural nets tend to be referential in nature.
If no limiting action is taken by the system's "manager" programs and the process is allowed to continue unchecked, it may lead to partial or total corruption of the entire system.
Lal suffered just such a cascade failure in 2366 when her neural net, its pattern copied from her father Data, proved unstable and unable to cope with poorly defined concepts such as emotion. (TNG: "The Offspring")
Cascade failure can also affect larger systems such as those on starships. (VOY: "The Haunting of Deck Twelve")
Cascade failure can also be more broadly defined to indicate a systemic failure originating in a specific area and then spreading via corrupted algorithms. In cases such as this the system in question need not be a neural net or even programmatic in nature. Such a cascade failure occurred in 2369 when Data initiated a plasma cascade failure in order to test the response of the exocomps to an original problem. (TNG: "The Quality of Life")