Flow slide is a flow-type landslide, which has high mobility and is destructive to the downstream infrastructures and buildings. In this study, a numerical method is presented to simulate the runout process of flow slide, and the interactions between flow slide and buildings on three-dimensional terrain considering the destruction process. In this proposed method, the flow slide is simulated by smoothed particle hydrodynamics (SPH) and the buildings are simulated by the finite element method (FEM). A node-to-surface contact algorithm is adopted to transfer displacements and contact forces between the flow slide and the buildings. An element erosion algorithm is adopted to simulate the destruction process. The method is validated using an analytical solution to dam break problem and the experimental results in two flume tests. A catastrophic flow slide involving building destruction is then simulated to further test the model performance. The flow depth, flow velocity, deposition area, and building damage degree are all reasonably simulated. The typical destruction process of buildings and patterns of the flow slide are also identified. The method will contribute to a better understanding of flow-structure interactions and is a promising tool for hazard analysis and mitigation.