Accelerometers embedded in smartphones have become an alternative means of measuring the roughness of roads. However, the differences in their sensitivity and sampling rates between smartphones could produce measurement inconsistencies that challenge the wide spread of the smartphone approach for road roughness measurements. In this study, the roughness measurement inconsistency was investigated between smartphones from three different brands. Using the same vehicle, device mount method, traversal speed, and method of producing a roughness index, field experiments demonstrated that accelerometer sensitivities and maximum sample rates vary significantly among smartphones of the same brand as well as across brands. For each smartphone, to achieve a margin-of-error within a 95% of confidence, significant large amounts of traversals are needed. Specifically, 24 and 35 traversals for a paved and an unpaved road, respectively. A higher sampling rate produced more consistent measurements and the least margin-of-error but resulted in larger data sizes. In addition, the measurements from all smartphones were not very sensitive to the size of the feature extraction window, therefore, selecting the largest practical window size will minimize the data size without significant loss of accuracy. For practical application, calibration is necessary to achieve consistent roughness measurements between various different smartphones.