Abstract: Concealed fault is an important geological problem in coal mine production. The scale of concealed fault is often small and it is difficult to predict accurately. In order to study the machine learning optimization algorithm suitable for predicting concealed faults, a prediction model for concealed fault density and a zonal evaluation method of fault complexity based on VMD-SABO-XGBoost are proposed, and the variational mode decomposition (VMD) algorithm is used to decompose the initial data such as gas content and firmness coefficient collected in the mine site into multiple modal components, and the feature extraction is carried out. The grey correlation analysis method is used to calculate the correlation degree ranking of the predicted features. Among them, the five characteristic indexes of fault strength parameters, fault fractal dimension, gas content, coal seam floor dip angle variation coefficient and firmness coefficient have the greatest correlation with fault occurrence. The elite reverse learning and adaptive weight factor are introduced into subtraction-average-based optimizer (SABO) optimization algorithm to solve the problems that the model falls into the “local optimal solution” prematurely, the convergence performance is poor and the diversity of the characteristic population is insufficient. The accuracy difference of the algorithm before and after optimization is verified by experiments. It is proved that the optimized SABO greatly improves the convergence efficiency and the prediction accuracy of the model. Combined with extreme gradient boosting (XGBoost) machine learning algorithm, a concealed fault prediction model is formed. Taking the fault of 3^# coal heading face in a mine in Changzhi, Shanxi Province as the experimental object, the comprehensive zonal evaluation grade of fault type is established, and it is used as a prediction label to compare with the traditional XGBoost model, VMD-XGBoost model and SABO-XGBoost model. The results show that the VMD-SABO-XGBoost model has the highest prediction accuracy. A prediction model for concealed fault density and a partition evaluation method of fault complexity based on VMD-SABO-XGBoost are proposed. The accuracy and adaptability of the method are verified by an example, which is corresponding to the actual fault complexity in the field. It can be applied to predict the density and complexity of concealed faults in mines.