Polyp classification is a feature selection and clustering process. Picking the most effective features from multiple polyp descriptors without redundant information is a great challenge in this procedure. We propose a multilayer feature selection method to construct an optimized descriptor for polyp classification with a feature-grouping strategy in a hierarchical framework. First, the proposed method makes good use of image metrics, such as intensity, gradient, and curvature, to divide their corresponding polyp descriptors into several feature groups, which are the preliminary units of this method. Then each preliminary unit generates two ranked descriptors, i.e., their optimized variable groups (OVGs) and preliminary classification measurements. Next, a feature dividing-merging (FDM) algorithm is designed to perform feature merging operation hierarchically and iteratively. Unlike traditional feature selection methods, the proposed FDM algorithm includes two steps for feature dividing and feature merging. At each layer, feature dividing selects the OVG with the highest area under the receiver operating characteristic curve (AUC) as the baseline while other descriptors are treated as its complements. In the fusion step, the FDM merges some variables with gains into the baseline from the complementary descriptors iteratively on every layer until the final descriptor is obtained. This proposed model (including the forward step algorithm and the FDM algorithm) is a greedy method that guarantees clustering monotonicity of all OVGs from the bottom to the top layer. In our experiments, all the selected results from each layer are reported by both graphical illustration and data analysis. Performance of the proposed method is compared to five existing classification methods by a polyp database of 63 samples with pathological reports. The experimental results show that our proposed method outperforms other methods by 4% to 23% gains in terms of AUC scores.