Since conventional in situ measurements of suspended sediments in the river system are labor-intensive and time-consuming, remote sensing approaches using multi- or hyper-spectral cameras have widely been applied to obtain high-resolution suspended sediment concentration (SSC) distributions in rivers and streams. However, in nature, the properties of heterogeneous sediment, such as the mineral content and particle size distribution, induce a strong variability in the optical images of the suspended sediments. For this reason, the robust estimation of the suspended sediment using the remote sensing technique is challenging due to the optical variability of the suspended sediment. Thus, it is necessary to deal with this variability of the optical images to improve the accuracy of remote sensing-based SSC measurements and extend them to the global estimator. In this study, a robust Machine Learning (ML) model for SSC estimation based on hyperspectral images was developed by considering the optical variability of the suspended sediment in water bodies. A series of field-scale tracer experiments were conducted in open channels with three different sediment types in order to obtain both the SSC using laser diffraction sensors and hyperspectral images using a UAV camera. The experimental results showed that the optical characteristics of SSC were critically heterogeneous due to the properties of the sediment. Using these experimental dataset, four explicit regression models and two implicit ML regression models were developed and compared to select an optimal estimator. Consequently, a Support Vector Regression (SVR) model using relevant spectral bands in a wide wavelength range yielded the most accurate results, with an R2 of 0.90 for the whole dataset. However, linear regression models, which could not consider various spectral bands and the nonlinear effect of the optical variability of SSC, were limited in their ability to retrieve SSC from hyperspectral images. Furthermore, the SVR model accurately reproduced the spatio-temporal SSC distributions in all study cases, including low-visibility suspended sediments, thus successfully resolved the optical variability of SSC with widely selected spectral bands from recursive feature elimination (RFE). The SVR model also successfully retrieved the SSC distribution in uncalibrated rivers. The results of this study demonstrated that the proposed ML regression models based on the hyperspectral imagery achieved a significant improvement in SSC estimation in terms of accuracy and global applicability.