Geollery: Social Media Metaverse in Virtual and Augmented Reality

We present Geollery, an interactive mixed reality social media platform for creating, sharing, and exploring geotagged information. Geollery introduces a real-time pipeline to progressively render an interactive mirrored world with three-dimensional (3D) buildings, internal user-generated content, and external geotagged social media. This mirrored world allows users to see, chat, and collaborate with remote participants with the same spatial context in an immersive virtual environment. We describe the system architecture of Geollery, its key interactive capabilities, and our design decisions. Finally, we conduct a user study with 20 participants to qualitatively compare Geollery with another social media system, Social Street View. Based on the participants' responses, we discuss the benefits and drawbacks of each system and derive key insights for designing an interactive mirrored world with geotagged social media. User feedback from our study reveals several use cases for Geollery including travel planning, virtual meetings, and family gathering.

Publications

teaser image of Geollery: A Mixed Reality Social Media Platform

Geollery: A Mixed Reality Social Media PlatformLive Demo of a Metaverse of Mirrored World

Ruofei Du, David Li, and Amitabh Varshney
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.
Keywords: metaverse, virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction, digital twins, mirrored world; digital world; digital world; augmented communication
teaser image of Social Street View: Blending Immersive Street Views With Geo-Tagged Social Media

Social Street View: Blending Immersive Street Views With Geo-Tagged Social MediaBest Paper Award

Ruofei Du and Amitabh Varshney
Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.
Keywords: metaverse, spatial-temporal virtual reality; social media; street view; geographical information systems; mixed reality; WebGL; digital twins; digital world
teaser image of Fusing Multimedia Data Into Dynamic Virtual Environments

Fusing Multimedia Data Into Dynamic Virtual EnvironmentsPh.D. Dissertation

Ruofei Du
Ph.D. Dissertation, University of Maryland, College Park., 2018.
Keywords: social street view, geollery, spherical harmonics, 360 video, multiview video, montage4d, haptics, cryptography, metaverse, mirrored world
teaser image of Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media

Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media

Ruofei Du, David Li, and Amitabh Varshney
Proceedings of the 24th International Conference on Web3D Technology (Web3D), 2019.
Keywords: virtual reality, mixed reality, 360° image, GIS, 3D reconstruction, projection mapping, mirrored world, social media, WebGL, metaverse, mirrored world, interactive graphics, digital world
teaser image of Experiencing a Mirrored World With Geotagged Social Media in Geollery

Experiencing a Mirrored World With Geotagged Social Media in Geollery

Ruofei Du, David Li, and Amitabh Varshney
Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA), 2019.
Keywords: virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction, metaverse, mirrored world
teaser image of Interactive Fusion of 360° Images for a Mirrored World

Interactive Fusion of 360° Images for a Mirrored World

Ruofei Du, David Li, and Amitabh Varshney
2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019.
Keywords: virtual reality, 360° image, 3D reconstruction, mixed reality, projection mapping, mirrored world, metaverse, mirrored world
teaser image of Systems, Devices, and Methods for Generating a Social Street View

Systems, Devices, and Methods for Generating a Social Street View

Ruofei Du and Amitabh Varshney
US Patent 10,380,726, 2016.
Keywords: virtual reality; social media; street view; geographical information systems; mixed reality; WebGL

Videos

Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media


Social Street View: Blending Immersive Street Views With Geo-Tagged Social Media


Talks

Interactive Perception & Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Perception & Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at UCLA by Prof. Yang Zhang , Remote Talk.


Interactive Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at ECL Seminar Series by Dr. Alaeddin Nassani , Remote Talk.


Interactive Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at Empathic Computing Lab , Remote Talk.


Interactive Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at UMD by Prof. Amitabh Varshney , College Park, Maryland.


Computational Interaction for a Universally Accessible Metaverse Teaser Image.

Computational Interaction for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at GAMES Seminar , Remote Talk.


Fusing Physical and Virtual Worlds into An Interactive Metaverse Teaser Image.

Fusing Physical and Virtual Worlds into An Interactive Metaverse

Ruofei Du

Invited Talk at UCLA by Prof. Yang Zhang , Remote Talk.


Polymerizing Physical and Virtual Worlds into  An Interactive Metaverse Teaser Image.

Polymerizing Physical and Virtual Worlds into An Interactive Metaverse

Ruofei Du

Invited Talk by Prof. Arthur Theil at Birmingham City University , Remote Talk.


Blending Physical and Virtual Worlds into  An Interactive Metaverse Teaser Image.

Blending Physical and Virtual Worlds into An Interactive Metaverse

Ruofei Du

Invited Talk at Wayne State University , Remote Talk.


Fusing Physical and Virtual Worlds into 
Interactive Mixed Reality Teaser Image.

Fusing Physical and Virtual Worlds into Interactive Mixed Reality

Ruofei Du

Invited Talk at George Mason University , Remote Talk.


Interactive Fusion of 360° Images for a Mirrored World Teaser Image.

Interactive Fusion of 360° Images for a Mirrored World

Ruofei Du

VR 2019, Osaka, Japan.


Cited By

  • A Systematic Literature Review of Mixed Reality Environments in K-12 Education. Education and Information Technologies.Nikolaos Pellas, Ioannis Kazanidis, and George Palaigeorgiou. source | cite | search
  • Digital Twin As a Mixed Reality Platform for Art Exhibition Curation. 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW).Inhwa Yeom and Woontack Woo. source | cite | search
  • GAVIN: A New Platform for Enriching 3D Virtual Indoor Navigation With Social-based Geotags. 26th IEEE Symposium on Computers and Communications (ISCC 2021).Christian Sicari, Valeria Lukaj, Antonio Celesti, Maria Fazio, and Massimo Villari. source | cite | search
  • What Was Hybrid? a Systematic Review of Hybrid Collaboration and Meetings Research. https://arxiv.org/pdf/2111.06172.pdf.Thomas Neumayr, Banu Saatci, Sean Rintel, Clemens Klokmose, and Mirjam Augstein. source | cite | search
  • Mapping HCI Research Methods for Studying Social Media Interaction: A Systematic Literature Review. Computers in Human Behavior.Yuya Shibuya, Andrea Hamm, and Teresa Cerratto Pargman. source | cite | search
  • Digitizing Intangible Cultural Heritage in China: A Pedagogical Model for Innovation and Entrepreneurship Among New Media Art Students. Lecture Notes in Computer Science.YunQiao Su, Sze Seau Jill Lee, BeiLe Su, and LiLi Zhang. source | cite | search
  • Improving Information Acquisition in City Tours Via Simplified Virtual Scenes With Location-Based POIs. Lecture Notes in Computer Science.Yujia Cao, Yiyi Zhang, and Tatsuo Nakajima. source | cite | search
  • A Virtual Reality Whiteboard System for Remote Collaboration Using Natural Handwriting. Electronics.Jiangkun Wang and Lei Jing. source | cite | search
  • Working Together Apart Through Embodiment. Proceedings of the ACM on Human-Computer Interaction.Guo Freeman, Dane Acena, Nathan J. McNeese, and Kelsea Schulenberg. source | cite | search
  • Visualization/AR/VR/MR Systems. Active Lighting and Its Application for Computer Vision.Katsushi Ikeuchi, Yasuyuki Matsushita, Ryusuke Sagawa, Hiroshi Kawasaki, Yasuhiro Mukaigawa, Ryo Furukawa, and Daisuke Miyazaki. source | cite | search
  • Geolocated Visual Summarization of Social Media Data. 2020 28th Signal Processing and Communications Applications Conference (SIU).Elif Sanlialp and M. Abdullah Bulbul. source | cite | search
  • Automatic Discovery and Geotagging of Objects From Street View Imagery. Remote Sensing.Vladimir Krylov, Eamonn Kenny, and Rozenn Dahyot. source | cite | search
  • Deep Depth Estimation on 360\textdegree Images With a Double Quaternion Loss. 2020 International Conference on 3D Vision (3DV).Brandon Feng, Wangjue Yao, Zheyuan Liu, and Amitabh Varshney. source | cite | search
  • Foveated Rendering Technique in Modern Graphics. University of Maryland, College Park.Xiaoxu Meng. source | cite | search
  • Apparatus, Method, and System for Presentation of Multimedia Content Including Augmented Reality Content. U.S. Patent 10,984,600.Leon Chen, Brian Keffer, and Jocelyn Seever. source | cite | search
  • An Antiquity to the Contemporary of Secret Sharing Scheme. Journal of Innovative Image Processing.K. Chitra and V. Prasanna Venkatesan. source | cite | search
  • Impact of Key Shape and Dimension on Text Entry in Virtual Reality. Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems.Tafadzwa Joseph Dube and Ahmed Arif. source | cite | search
  • Virtual and Augmented Reality Applications in Science and Engineering. Computing in Science & Engineering.Nargess Memarsadeghi and Amitabh Varshney. source | cite | search
  • Leveraging the Physical Properties of Real Objects to Manage Digital Photography in Augmented Reality. Journal of KIISE.H. Chae, Youli Chang, Minji Kim, Gwanmo Park, and Jinwook Seo. source | cite | search
  • Students' Expectations and Social Media Sharing in Adopting Augmented Reality. The International Journal of Information and Learning Technology.Ari Alamaki, Amir Dirin, and Jyrki Suomala. source | cite | search
  • Surveying Consumer Understanding \& Sentiment of VR. Proceedings of the International Workshop on Immersive Mixed and Virtual Environment Systems (MMVE '21).Joseph O'Hagan, Mohamed Khamis, and Julie Williamson. source | cite | search
  • Culture Meets Immersive Environments: A New Media Landscape Across Europe. Universitat Autònoma de Barcelona.Marta Brescia-Zapata. source | cite | search
  • Exploring the Effectiveness of Face-to-face Mixed Reality for Teaching With Chalktalk. https://arxiv.org/abs/1912.03863.Zhenyi He and Ken Perlin. source | cite | search
  • Video Content Representation to Support the Hyper-reality Experience in Virtual Reality. 2021 IEEE Virtual Reality and 3D User Interfaces (VR).Hyerim Park and Woontack Woo. source | cite | search
  • A Survey on Synchronous Augmented, Virtual and Mixed Reality Remote Collaboration Systems. ACM Computing Surveys.Alexander Schäfer, Gerd Reis, and Didier Stricker. source | cite | search
  • Interactive Social Media Vsualization. Ankara Yildirim Beyazit University.Elif Sanlialp. source | cite | search
  • Teleportation Links for Mixed Reality Environments. U.S. Patent 10,831,334.Paul Simon Martin. source | cite | search
  • Telelocation: Location Sharing for Users in Augmented and Virtual Reality Environments. U.S. Patent 10,403,044.Paul Simon Martin. source | cite | search
  • Fine-grain Placement and Viewing of Virtual Objects in Wide-area Augmented Reality Environments. U.S. Patent Application 15/600,449.Paul Simon Martin. source | cite | search
  • Media Tags Location-anchored Digital Media for Augmented Reality and Virtual Reality Environments. U.S. Patent Application 15/600,446.Paul Simon Martin. source | cite | search
  • Methods, Systems, and Media for Enhancing Two-dimensional Video Content Items With Spherical Video Content. U.S. Patent Application 15/366,432.Leon Bayliss and Richard Hale. source | cite | search
  • Six-degree of Freedom Video Playback of a Single Monoscopic 360-degree Video. U.S. Patent 10,368,047.Che Chen, Zhil Zhili, Duygu Ceylan Aksit, Jingwei Huang, and Hailin Jin. source | cite | search
  • Impression Tagging System for Locations. U.S. Patent 10,545,996.David Grochocki and Silvana P. Moncayo Torres. source | cite | search
  • Augmented Reality Messaging Syste. U.K. Patent GB2588838A.Wang Yue. source | cite | search
  • Information Processing Apparatus, Object Recognition Apparatus, Device Control System, Movable Body, Image Processing Method, and Computer-readable Recording Medium. U.S. Patent 10,937,181.Aman Amano, Seiy Seiya, Hiroyoshi Sekiguchi, and Soichiro Yokota. source | cite | search
  • Computer Vision System That Provides Information Relative to Animal Wellness and Habitat/intervention Design. U.S. Patent 10,943,102.Mark Raymond Miller, Archana Ramachandran, Christopher C. Anderson, and Aaron Spafford. source | cite | search
  • Generating Three-dimensional Content From Two-dimensional Images. U.S. Patent 11,113,887.Daniel Kopeinigg, Arthur Van Hoff, Philip Lee, Solmaz Hajmohammadi, Sourabh Khire, Simion Venshtain, and Andrew Walkingshaw. source | cite | search
  • Generating Immersive Trip Photograph Visualizations. U.S. Patent 11,113,882.Michal Luka, Zhili Chen, Jan Brejcha, and Martin Cadik. source | cite | search
  • Immersive Street-level Social Media in the 3D Virtual City. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing.Hannu Kukka, Minna Pakanen, Mahmoud Badri, and Timo Ojala. source | cite | search
  • Instant Panoramic Texture Mapping With Semantic Object Matching for Large-Scale Urban Scene Reproduction. IEEE Transactions on Visualization and Computer Graphics.Jinwoo Park, Ik-Beom Jeon, Sung-Eui Yoon, and Woontack Woo. source | cite | search
  • Immersive Trip Reports. Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology.Jan Brejcha, Michal Lukác, Zhili Chen, Stephen DiVerdi, and Martin Cad{\'{\i}}k. source | cite | search
  • Social Media Based 3D Visual Popularity. Computers & Graphics.Abdullah Bulbul and Rozenn Dahyot. source | cite | search
  • A Review on Mixed Reality: Current Trends, Challenges and Prospects. Applied Sciences.Somaiieh Rokhsaritalemi, Abolghasem Sadeghi-Niaraki, and Soo-Mi Choi. source | cite | search
  • A Review of Mixed Reality Telepresence. IOP Conference Series: Materials Science and Engineering.Fazliaty Edora Fadzli, Ajune Wanis Ismail, Mohamad Yahya Fekri Aladin, and Nur Zuraifah Syazrah Othman. source | cite | search
  • Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis From a Single Image. 2021 IEEE/CVF International Conference on Computer Vision (ICCV).Ronghang Hu, Nikhila Ravi, Alexander C. Berg, and Deepak Pathak. source | cite | search
  • Deep Learning-based Semantic Segmentation of Urban-scale 3D Meshes in Remote Sensing: A Survey. International Journal of Applied Earth Observation and Geoinformation.Jibril Muhammad Adam, Weiquan Liu, Yu Zang, Muhammad Kamran Afzal, Saifullahi Aminu Bello, Abdullahi Uwaisu Muhammad, Cheng Wang, and Jonathan Li. source | cite | search
  • Co-Speculating on Dark Scenarios and Unintended Consequences of a Ubiquitous(ly) Augmented Reality. Proceedings of the 2023 ACM Designing Interactive Systems Conference.Chloe Eghtebas, Gudrun Klinker, Susanne Boll, and Marion Koelle. source | cite | search
  • Mixed Reality: The Interface of the Future. Technical Report.Dipesh Gyawali. source | cite | search
  • Augmented Reality Content Generator for Suggesting Activities at a Destination Geolocation. US Patent App. 17/146,973.Kaveh Anvaripour, Ilteris Kaan Canberk, Virginia Drummond, Jean Luo, Alek Matthiessen, and Celia Nicole Mourkogiannis. source | cite | search
  • Mixed Reality in Education: A Bibliometric Analysis of Ten Years of Research. Proceedings of the 4th International Conference on Virtual Reality.Das Runi Mani and J. V Madhusudan. source | cite | search
  • 360RVW: Fusing Real 360\textdegree Videos and Interactive Virtual Worlds. Proceedings of the 31st ACM International Conference on Multimedia.Mizuki Takenawa, Naoki Sugimoto, Leslie Wöhler, Satoshi Ikehata, and Kiyoharu Aizawa. source | cite | search
  • Building Social Presence for Transnational Families Through Mixed Reality Shared Experiences. 7.Shuting Zhou, Alexis Morris, and Sara Diamond. source | cite | search
  • Using Google My Maps to Collect Vernacular Toponyms. Onomastics in Interaction With Other Branches of Science Volume 1 Keynote Lectures Toponomastics.Sara Racca. source | cite | search
  • Stay In Touch