Meet the project: 3DforXR
Can you briefly explain what your project is all about? What’s unique about it?
3DforXR is about generating 3D assets, i.e. photorealistic 3D models, in a fast, easy and straightforward way, processing them, and then using them inside eXtended Reality applications and experiences. By combining 3D Computer Vision and Artificial Intelligence, 3DforXR will support three different modalities. 1) multi-view 3D reconstruction, where one can upload multiple overlapping images of an object and obtain automatically an exact 3D copy of it, 2) single image 3D prediction, where a single front-facing image of an object is enough to generate a 3D model that resembles, as well as possible, to reality and 3) 3D from text, where one provides just a textual description of the 3D model he needs for an XR application and 3DforXR tool generates a relative 3D asset. Although several software solutions for 3D reconstruction exist in the market, 3DforXR’s unique point is that it offers a single point, a web application at the end of the project, that combines different approaches, where one can generate 3D models optimized and ready to be used in XR applications, from different inputs, images or just text.
What’s the biggest milestone with your project your startup(s) have achieved so far, and what has surprised you most on this journey?
We are currently very close to the completion of the second release of the 3DforXR project which corresponds with the deployment of the enhanced versions of the two modalities that correspond to 3D asset generation from multiple or single images. We are excited by the progress achieved so far and were surprised in a positive way by the amelioration of the results in the demanding single image 3D prediction module. A library with processing tools to modify both the geometry and the appearance of the derived 3D models is also ready to be shared with SERMAS partners and we can’t wait for their valuable feedback on the second release of our tools.
How did you measure success?
To measure the success of the two developed modalities we performed quantitative and qualitative evaluations of their results. To evaluate the accuracy of the generated 3D models two KPIs were estimated. For the multi-view 3D reconstruction approach, we measured depth estimation accuracy which was higher than 85%. To estimate this, we run our software on synthetic data, i.e. images synthesized through computer graphics from ground-truth 3D models and compared the estimated depth maps corresponding to each image against the ground truth ones. For the single-image prediction module a similar approach was followed, where instead of depth maps, we compared the predicted 3D models against a publicly available dataset. An F-Score of 80% was achieved for the successful examples.
What are your goals over the next three and six, months?
Our main goal for the next three months is to focus on the development of the third modality of 3DforXR, which is the generation of 3D assets from textual descriptions. Looking further ahead in time in the next 6 months we are looking forward to integrating all the 3DforXR technology in the SERMAS toolkit, offering it to end users via a single web application with an intuitive User Interface and proceeding with actions towards the communication, dissemination and exploitation of the project outcomes. A preparation of a publication to share our technological progress with the scientific XR and Computer Vision community is also in our goals towards the last trimester of the project.
How has SERMAS helped you during the past few months?
Besides the obvious help of providing valuable funding to develop 3DforXR, mentoring meetings helped us verify that we are on the right track, while the feedback from the evaluation of the first Release allowed our team to focus on what is considered more important by the SERMAS consortium in order to maximize the impact of our solution.
Company: Up2metric
