COMPARISON OF YOLOV8 AND PYTORCH-RETINANET FOR VEHICLE DETECTION

Abstract

This study aims to evaluate and compare the effectiveness of two deep learning models - PyTorch-RetinaNet and YOLOv8 - for vehicle detection, addressing the challenges in object detection across varying size, shape, and lighting conditions. The research methodology utilized a comprehensive dataset of 4,058 vehicle images with 12 distinct object classes, implementing both models with varying learning rates (0.001, 0.01, and 0.0001). The dataset was split into training (65%), validation (24%), and testing (11%) sets, with preprocessing techniques including image resizing, brightness normalization, and data augmentation applied to enhance model performance. The experimental results revealed distinct capabilities for each model: PyTorch-RetinaNet achieved a mAP50 of 38.6% and mAP50-95 of 24.7%, exhibiting particular strength in detecting large objects (mAP50-95 of 42.0%) and maintaining stable recall metrics (AR@1: 30.9%, AR@10: 54.7%, AR@100: 55.9%). In contrast, YOLOv8 demonstrated superior overall performance with a mAP50 of 45.6%, mAP50-95 of 33.0%, precision of 48.3%, and recall of 61.5%, particularly excelling in handling overlapping objects with confidence scores of 0.79-0.89. The findings suggest YOLOv8 is more suitable for real-time applications, while PyTorch-RetinaNet excels in scenarios requiring precise detection across varying object sizes.