Gen-AI In Metaverse

This is the repository of Gen-AI In Metaverse project, developed for the "Advanced User Interfaces" Course Project (prof. Franca Garzotto and tutor Alberto Patti) @ Politecnico di Milano, a.a. 2023/2024 evaluated with 30/30 CumLaude

Overview

Gen-Ai in Metaverse is a VR application, integrated with Gen-Ai technology that addresses social anxiety by providing a secure environment for users to enhance their social skills. Virtual companions, crafted with empathy, facilitate realistic conversations to aid users in overcoming social challenges. The platform utilizes Gen-Ai to create personalized, friendly avatars, enhancing the interactions within a realistic virtual setting. Real-time communication is enabled through Azure, while the interactions of each avatar are controlled by GPT APIs

How To

To run the project on your machine, you need to have Unity3D installed on your machine, and to follow the instructions below:

1. create a .env file in the main folder (before Assets/ folder) containing the following parameters:
   • AzureKey with the key for Azure Cognitive Speech Service;
   • Region with the selected region for Azure Cognitive service;
   • OpenAI with the token for the connection to the OpenAI API;
2. Open Unity and import the package SpeechSDK from Microsoft Cognitive Service, you can find it here.
3. To try the application you can either build the APK with Unity (the application has been tested with Meta Quest 2).
4. Or you can try the application directly from the editor plugging you VR device in.

Video Demonstration

Credits

• Lorenzo Corrado
• Roberto Cialini
• Filippo Pantaleone
• Chukun Zhang