Object detection and tracking in maritime videos are crucial for marine management today. Nevertheless, due to the small size of ships in maritime video, current state-of-the-art algorithms are incapable of producing satisfactory results. We design an improved YOLOX (iYOLOX) and DeepSORT-based method to detect and track small objects in maritime videos to improve detection accuracy. Initially, the primary feature extraction capability of YOLOX’s backbone network is improved by adding the convolutional block attention mechanism. Second, we employ the context enhancement module to enlarge the receptive field while preserving the specific features. Finally, a parallel deformable convolution bifusion feature pyramid network is utilized for bidirectional feature extraction for small objects. iYOLOX is additionally combined with the DeepSORT algorithm to track targets. The experimental results show iYOLOX performed better when compared with the original YOLOX such that iYOLOX improves 3% average precision (AP) on the COCO2017 dataset, 4.7% AP_s for small-size objects, and 3.41% AP₅₀ on the LMD-TShip maritime video dataset. Furthermore, it indicates that the proposed algorithm effectively alleviates some object occlusion and camera shaking issues and improves the model’s utility and robustness.