Hard exudate is one of the early symptoms of diabetic retinopathy (DR). If hard exudates are detected promptly, the doctor can provide timely treatment advice for the patient’s condition. However, due to the specific nature of hard exudates in the fundus, there are significant differences in the size of the lesions. To solve this problem, a dual-branch transformer network is proposed, which can effectively balance the size difference of lesions and improve the segmentation performance. First, the image is segmented into small pieces of the same size and the same image is fed into two branches. Then, the features are extracted with CNN, trained with transformer, sampled on the decoder, and fused with the features obtained from CNN. The segmentation of lesions is improved by adjusting the lesion sizes and their weights between different branches to gradually focus on small lesions. This method is tested on the e-ophtha and IDRiD datasets to evaluate the network performance. The results show that our method can achieve better performance than the most advanced methods at present.