Abstract: The cusp mutation model of the bending failure of layered rock mass (rock beam) based on the geological mechanical model and the potential energy function is proposed, that is used in the study on outside control condition when mutation happens to system state, and mainly expatiates on how nonlinear system tends towards system character mutation from continuous gradual change state. The mechanism of releasing energy in mutation process was analyzed by using this theory. According to the function of stationary curved surface and branch curve, the necessary-sufficient mechanical conditions of system instability were derived. The method and the instability criterion are developed to analyze the stability of layered rock mass. The instability criterion of rock beam reveals that the stability of layered rock mass is controlled by the external force p，q, the modulus of elasticity E, the severe of rock mass r_c, the interlayer shear stress τ and the geometry l，h. Combined the mine example, the consistency between applying cusp mutation criterion for instability analysis and the actual results is more than 84%.