The slope failure is a large deformation problem, the traditional numerical method finite element method (FEM), and limit equilibrium method (LEM) is difficult to characterize the whole process of the failure, resulting in the mechanism analysis which is not clear. In this work, a mesh-free particle approach, named the smoothed particle hydrodynamics (SPH) method, is presented and is improved to analyze the damage and failure process of a rock slope. In order to better describe the cause and mechanism of brittle failure for rock slope, the plastic factor was suggested and introduced into the SPH algorithm, and the conservation equations of SPH for brittleness characteristics were obtained. For soil slope, its instability mechanism and failure mode are different from that of rock slope. Therefore, this work also introduces the application of the smoothed particle hydrodynamics (SPH) method in the instability evolution process of soil slope. Firstly, an improved method for solving the SPH safety factor based on the adaptive discount factor method is proposed, the tolerance value is introduced, the relationship between the displacement increment and the tolerance value of the characteristic point is analyzed, and the real displacement mutation point is determined. Secondly, the SPH numerical model of slope is established to analyze the characteristics of slope deformation and failure, which shows that the SPH method can effectively represent the whole process of slope instability and progressive failure and effectively overcome the disadvantage of non-convergence when the traditional numerical method is used to study the deformation and failure of geotechnical materials.