We present a stereo matching approach referred to as HLocalExp-CM by exploiting the hierarchical local contextual information and a confidence map based on a new grid structure. The proposed approach preserves fine depth edges and extracts accurate disparities in weak texture, textureless, and repeated texture regions. The proposed approach adopts a two-stage optimization strategy. In the framework of first stage, a multiresolution cost aggregation is minimized to reduce the search space of the disparity plane of each pixel. The second stage iteratively optimizes the confidence map and a global energy function to progressively improve the disparity accuracy for each pixel. The confidence map is estimated through classifying the pixels into distinctive and ambiguous ones by computing the decreasing rate of the multiresolution cost aggregation and then performs a spatial propagation and plane refinement for the update of the disparity of each pixel, thereby successfully eliminating the ambiguity of nondistinctive pixels. The global energy function based on a pairwise Markov random field uses cross-scale cost aggregation for taking advantage of context information of objects in different scenarios on local grid regions, which is different from the deep learning technique uses convolution layers extracting the context information. The proposed approach is evaluated on Middlebury benchmark V3, and is ranked first based on “bad 2.0 all metric,” a widely used criterion for the evaluation of stereo images, while the eighth place on “bad 2.0 nonocc metric” (recorded on July 24, 2021).