How to make a mobile (flutter) app that also works in VR (unity)

Question

I'm interested to build a VR app which will allow the user to interact with some 2d screens in a 3d world.

I would like these same 2d screens to be available on a mobile app, so that the user could use them either from their iPad, or inside the VR too.

My first thought was to create the ui in flutter and build the apps for the 2d, but there seems to be no way to actually get flutter elements into either unreal engine or unity.

I'm wondering what would be the best way to achieve this without having to reinvent the wheel between VR and 2d Mobile apps?

Would it be better just to code the whole thing in unreal engine? And then grab the 2d elements out of it and use it to build a separate 2d app for mobile in unreal engine as well?

Or would it be best to just build two apps? one for mobile with flutter and one for unreal engine, where the 2d screen layouts are recreated in each app? I was kinda hoping to avoid this way because it will be hard to make everything look the same and will take more work in building multiple apps.

Answer 1

If you want this to actually succeed you need to nail down what user interface primitives actually work in both environments. Using a drop down on an iPad is very different than in VR.

Once you know what UI primitives work well in both carefully design your app to only rely on those primitives. Now ask yourself if this is enough to get work done.

If it's not then it's going to be two apps. Or at least one app that can tell what it's running on. Because now the UI has to adapt to the environment.

That said, it is possible to lay things out in similar ways in both just to help people find things. New users spend a lot of time just poking around learning where you put stuff. Once found they don't like it to move. Keep things in the same place on both or users will be calling asking where feature x went.

Answer 2

Why not have both apps be one in the same. Write both in Unity. You could render one “app” as a world-space canvas put on a model of an iPhone/whatever in VR. Then share that exact code with another project that renders to screen space. This second project can be built on mobile.

You will likely have to build out some way to do input in your vr “app”on the fake iPhone.