In the AI of the beholder: A comparative analysis of computer vision-assisted characterizations of human-nature interactions in urban green spaces

Big data from photo-sharing platforms offer unique opportunities for the study of human-nature interactions and landscape planning. Research increasingly relies on computer vision in artificial intelligence to identify elements of interest in photographs and user preferences and sentiment towards them. Studies largely rely on pre-trained models from one of several available cloud-based, commercial image recognition services, but the extent to which findings depend on the implemented technology has not yet been explored. Here, we analyze ~ 10,000 outdoor photographs retrieved from three social media platforms and geolocated within green and blue spaces in Haifa (Israel) by means of machine tags from three popular cloud-based services. We find that clustering of the 45 investigated sites based on common characteristics of the photographs is considerably affected by the image recognition service chosen, especially for sites with limited data points (<80 photographs). Moreover, after associating the individual tags to specific aspects of the outdoor experience, we find substantial differences in the identification and ranking of outdoor recreational activities, characterization of the local biophysical environment (e.g., wildlife and vegetation), and feelings associated with the photographs. With no image recognition service clearly outperforming the others in all evaluation criteria, we argue that the optimal choice of image recognition service to rely on likely depends on the intended final application. Time and resource permitting, future studies should consider combining information from multiple sources for a characterization that is more nuanced and less prone to be affected by the idiosyncrasies of the individual technologies.

Automated summary

Big data from photo-sharing platforms provide unique opportunities for studying human-nature interactions and landscape planning. This study analyzes ~10,000 outdoor photographs from three social media platforms to assess the impact of different image recognition services on clustering of photograph characteristics and other factors. The findings demonstrate that the choice of image recognition service significantly affects the results, highlighting the need for careful consideration in selecting the most appropriate service for specific applications.