The smoothness of smartphone operations is essential to user perceptual experience. However, the underlying mechanism of how smoothness can impact user experience has not been elucidated. In this paper, we conducted two experiments to explore factors that may potentially affect user perceptual experience of smoothness in smartphone operation and examined how these factors contributed. In experiment 1, ten participants were invited and they were sensitive to the smoothness of the smartphone. Participants used swiping gesture to manipulate 27 sequences with different forms of frame loss on the microblog simulation interface, and gave mean opinion scores (MOS) according to self perception. According to the experimental results, three factors that affect smoothness perception were found: single frame-loss number (SFLN), frame-loss time (FLT), and frame-loss interval (FLI). But the effect of each factor was effective only in some conditions. In experiment 2, 20 participants gave their MOS of 84 sequences, and these sequences were divided into three parts by three factors of experiment 1. Participants’ electrophysiological data was also collected to verify the validity of the participants’ scores. The results of an analysis of variance and Student–Newman–Keuls (SNK) test showed that a SFLN, FLT, and their interaction results significantly affected the user perception of smoothness in smartphone operation. Specifically, the effect of a SFLN was more significant than that of FLT. The effect of the FLI on user perceptual experience was relatively low, but the interactive effect of FLT and interval was significant. Finally, regression analyses were conducted to obtain the fitting formulas. Our research results reflect some preliminary suggestions, so as to guide developers to configure the performance of smart phones, in order to control the frame loss of smart phones and avoid bad user evaluation.