Finite element (FE) modeling of rough surfaces is becoming increasingly common. However, the quality of the assumptions being made in these models, and thus the quality of the models themselves, is often unclear. Decisions about the geometry of the surface to be modeled, including the size of the surface to be modeled, the lateral resolution of the measured surface data to be used, and the formulation of the probabilistic surface to be used, can have a significant effect on a model's behavior. Similarly, varying model parameters, including the FE mesh density, can change the results by a factor of three or more. This work examines some of the metrics that can be used to evaluate the influence of these assumptions and parameters on FE models with rough surfaces and discusses the relative merits of each option. In particular, qualitative comparison of result plots, quantitative comparison and convergence of results parameters, qualitative and quantitative comparison of distributions of result values over various model dimensions, and more sophisticated comparison techniques inspired by image and signal processing are discussed. SCANNING 33: 353-369, 2011. (C) 2011 Wiley Periodicals, Inc.