Edge-preserving image smoothing is essential for computational imaging. Traditional filters are based on low-level image gradients; however, the edges defined on gradients do not align with the contours of human perception, thus suffering from certain limitations. We propose an image filter based on a soft clustering model that combines high-level semantics (derived from instance segmentation) with low-level features (intensities). The proposed filter works by first performing soft clustering on the input image according to the high- and low-level features to derive the affinity matrices, which are then fused for image smoothing. Experiment results indicate the advantages of the proposed filter in a variety of applications, including image smoothing, flash/non-flash fusion, detail enhancing, image dehazing, and depth upsampling. Furthermore, the proposed filter is efficient, and it takes 1.75 s to process a color image with 1 megapixel on a modern desktop.