VISION-LANGUAGE SYSTEM FOR ASSISTING VISUALLY IMPAIRED INDIVIDUALS IN OBJECT RECOGNITION AND DESCRIPTION QUERY
Keywords
DOI:
https://doi.org/10.34238/tnu-jst.14879Abstract
Individuals with visual impairments encounter substantial difficulties in recognizing and discerning objects within their environments due to the variability and intricacy of real-world settings, which significantly restrict their personal and social engagements. This study presents a scene perception support system for the visually handicapped, utilizing a large vision-language model (LVLM) to provide interaction via basic activities, including key pushes, screen touches, and voice instructions. The system provides a panoramic description while allowing users to query specific objects through accurate image segmentation based on the visual model. To overcome common optical illusions in visual-language models, the proposed method integrates image segmentation information as a supplementary knowledge source for multimodal inference, thereby improving the accuracy of scene perception. Experiments on the Visual Genome Regions dataset showed that the proposed system significantly improved its ability to identify and describe objects compared to the Qwen2-VL-Chat background model, as well as the LLaVA and BLIP-2 large language models, confirming its potential application in accessibility systems for the visually impaired.Downloads
References
M. Zou, A. Chen, Z. Liu, et al., “The burden, causes, and determinants of blindness and vision impairment in Asia: An analysis of the Global Burden of Disease Study,” Journal of Global Health, vol. 14, 2024, doi: 10.7189/jogh.14.04100.
D. L. Demmin and S. M. Silverstein, “Visual impairment and mental health: Unmet needs and treatment options,” Clinical Ophthalmology, vol. 14, pp. 4229-4251, 2020, doi: 10.2147/OPTH.S258783.
M. Cardona, K. Alwenya, A. Ur Rehman, et al., “Eye care interventions that reduce access inequities for women, rural residents and older people in low- and middle-income countries: A scoping review,” Frontiers in Public Health, vol. 13, 2025, doi: 10.3389/fpubh.2025.1578848.
A. Karamolegkou et al., “Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users,” in Proc. 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025, pp. 25949-25982, doi: 10.18653/v1/2025.acl-long.1260.
Y. Tian et al., “CAT+: Investigating and Enhancing Audio-Visual Understanding in Large Language Models,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 47, no. 10, Jun. 2025, doi: 10.1109/TPAMI.2025.3582389.
Q.-H. Nguyen et al., “Naviblind: A Multimodal AI Assistant for Visually Impaired Users to Identify Product Information from Images and Speech,” CTU Journal of Innovation and Sustainable Development, vol. 17, pp. 97-105, 2025, doi: 10.22144/ctujoisd.2025.057.
F. Tang et al., “Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 26147-26159, doi: 10.1109/cvpr52734.2025.02435.
J.-H. Kim, S.-W. Park, J.-H. Huh, S.-H. Jung, and C.-B. Sim, “Human Scene Understanding Mechanism Based Image Captioning for Blind Assistance,” IEEE Access, vol. 13, pp. 81933-81947, 2025, doi: 10.1109/ACCESS.2025.3564991.
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, and L.-J. Li, “Visual Genome: Connecting language and vision using crowdsourced dense image annotations,” Int. J. Computer Vision (IJCV), vol. 123, no. 1, pp. 32-73, 2017, doi: 10.1007/s11263-016-0981-7.
Qwen Team, “Qwen2-VL: A Large Vision Language Model,” arXiv preprint, arXiv:2409.12191, 2024, doi: 10.48550/arXiv.2409.12191.
A. Kirillov et al., “Segment Anything,” in Proc. IEEE Int. Conf. on Computer Vision (ICCV), 2023, pp. 4015-4026, doi: 10.1109/ICCV51070.2023.00371.
S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. S. Torr, and L. Zhang, “Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 6881-6890, doi: 10.1109/CVPR46437.2021.00681.
M. Oquab et al., “DINOv2: Learning Robust Visual Features without Supervision,” Trans. Machine Learning Research (TMLR), 2024, doi: 10.48550/arXiv.2304.07193.
H. Caesar, J. Uijlings, and V. Ferrari, “COCO-Stuff: Thing and stuff classes in context,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 1209-1218, doi: 10.1109/CVPR.2018.00132.
Y.-Y. Tseng, A. Bell, and D. Gurari, “VizWiz-FewShot: Locating Objects in Images Taken by People with Visual Impairments,” in Proc. Eur. Conf. on Computer Vision (ECCV), 2022, pp. 575-591, doi: 10.1007/978-3-031-20074-8_33.
