Most of the existing Siamese network-based trackers achieve the similarity matching problem through the cross-correlation of convolutional features between the template branch and the search branch. However, the method of using the overall feature of the target as the convolution kernel to cross-correlate with the search area contains a lot of background information, which will adversely affect the tracking. To solve this problem, we propose a feature fusion and saliency-attention Siamese network (MSSiamCAR) for object tracking, adopting a part-based tracking strategy to track discriminative local salient regions in objects. This algorithm uses the ResNet-50 network to extract the target features. To solve the loss of the shallow part of the target due to the deepening of the network, a multilayer feature is proposed. The fusion module enhances the feature extraction ability of the network for the target. Second, a saliency capture module is proposed to obtain the local saliency of the target. These saliency are robust to interference factors. The saliency interaction module is designed, and the graph attention mechanism is used to establish an effective connection between the saliency, and the target feature information is propagated to the search feature so that the tracker has more discriminative ability to the target feature. Finally, the region proposal network is used to perform operations, such as classification, regression, and centrality calculation. Experiments on challenging benchmarks, including OTB-100, VOT2018, large-scale single object tracking (LaSOT), generic object tracking-10k (GOT-10k), and unmanned aerial vehicles 123 (UAV123), demonstrate that our proposed tracker performs favorably against state-of-the-art trackers.