A cascade failure occurred when a "learning" program was corrupted, either by repeated application to a task it was ill suited to perform or by external influence such as damage to the hardware used to implement it.
In a neural net system, in particular a positronic brain, in which most or all sub-programs interacted, this could lead to corruption of surrounding algorithms and routines as they attempted to use data generated by the faulty application to perform a task of their own. This could compound the damage to the original sub-program, as neural nets tended to be referential in nature.
If no limiting action was taken by the system's "manager" programs and the process was allowed to continue unchecked, it could lead to partial or total corruption of the entire system.
Lal suffered just such a cascade failure in 2366 when her neural net, its pattern copied from her father Data, proved unstable and unable to cope with poorly defined concepts such as emotion. This resulted in total system failure – effectively death. (TNG: "The Offspring")
Cascade failure could also affect larger systems such as those on starships. The Talaxian freighter Salvoxia was lost to a cascade failure in the late 23rd century. (VOY: "The Haunting of Deck Twelve")
Cascade failure could also be more broadly defined to indicate a systemic failure originating in a specific area and then spreading via corrupted algorithms. In cases such as this the system in question did not need to be a neural net or even programmatic in nature. Such a failure occurred in 2369 when Data initiated a plasma cascade failure in order to test the response of the exocomps to an original problem. (TNG: "The Quality of Life")