Using Visbit to provide the highest-possible display quality for 360° videos
A problem begging to be solved.
As challenging as it is to create super high-quality 360° video (using the best camera and post-production techniques), it’s even more difficult to easily distribute content in the highest-resolution possible across multiple device types; especially to collaborators and customers with no experience in the 360° video ecosystem.
Here’s an example… I’m currently working on a project with a collaborator 3000 miles away and we’ll be going through multiple iterations of rough cuts. He’s not super technical, never used a headset before, and I gave him an Oculus Go to view the material. How am I remotely going to teach him how to download big files from dropbox and sideload them, including managing all the various versions of rough cuts I'll be sending him. How's he going to manage all those files on the device when he goes beyond his limit of RAM? Like a lot of my collaborators, he’s an artist, not a techno-geek. What are my options for pushing new drafts of videos out to him quickly without any friction and pain?
Plus, I’d prefer him (and all of my collaborators and clients) to see the 7k videos that I’m shooting on the Z CAM V1 in their best possible resolution instead of having to make and side-load low-res 4k versions. I’m viewing my edits on an HTC Vive Pro and would like everyone on my team to see the material in similar quality.
Is there a solution?
It seems as if this would be impossible, right? Well, not exactly…. There’s a way. Visbit has a technology ecosystem based on a technique they call “Visbit View Optimized Streaming.” VVOS delivers high-resolution imagery to only the portion of the 360° sphere where the viewer is looking; and low resolution imagery everywhere else. As the viewer turns their head, the Visbit player loads the high-res imagery into the new viewing area quickly (with only 20ms latency). This method of foveated rendering naturally requires less bandwidth to stream the scene to the device, even further enhanced by Visbit's development of a method to adjust the bitrate during streaming to dynamically adapt to each client’s bandwidth. This essentially predicts how bitrates change based on bandwidth... a novel technique for attempting to maximize image quality.
Hugh Hou helps explain complex stuff for us... again!
But all the above wouldn’t amount to much without the core technology Visbit uses to downsample high-resolution imagery to the headset. CY Zhou (a founder of Visbit and their CEO) explains this very well in this video featuring CY in a discussion with Hugh about video resolution and oversampling.
In short, when you’re looking at an arbitrary spot on the 360° sphere in the headset, the player has to determine the best possible color to render every pixel. Because your head may be tilted at an arbitrary angle, there's no 1:1 matching between the pixels of the source video and where those same pixels lie in the small area of the sphere that you’re seeing in the headset. It's easy to get yourself into a situation in which you're looking at a pixel that isn't fully covered by the source data, which forces the player to render that pixel in a way that's perceived as blurry. Higher resolution source video provides for better downsampling quality of the video because there's more information to determine the final pixel color, which maintains sharpness. This all happens with Visbit in real time as you rotate your head within the 360° sphere and the end effect is that low-cost headsets like the Oculus Go produce better quality imagery than if you’d simply side-loaded a 4k video or (even worse) streamed the video on Youtube (or VeerVR, for example). Visbit’s encoding process allows for this optimal method of rendering video to the output device and although both Facebook and Oculus have published tech papers in the view-dependant domain, the fact that Visbit has a complete ecosystem in place to support this rendering technique provides incredible value in the world of 360° video players. Here's a chart of the turnkey system that Visbit offers to content creators right now.
Let's not forget audio. Sound is an incredibly important element and the Visbit player supports ambisonic b-channel spatial audio correctly and seamlessly. Plus, if there’s no ambisonic track the player does a very cool thing.. it “spatializes” the audio track so that when the viewer turns their head, the audio rotates accordingly. This is useful (!) and gives my collaborators a sense of the spatial audio track even before I’ve provided the final mix.
One final thing to note about 360 players on the Go. I’m currently frustrated with the Oculus Gallery player because with my side-loaded videos I’m experiencing audio latency, making it seem as if the soundtrack is slightly out of synch with the video. And in the Samsung VR player the audio synch is fine but there’s a weird subtle flickering in the image that's incredibly annoying and makes people think that my source video is flickering. Argh. With the Visbit player I can be sure that my collaborators and clients see nearly exactly what I’m seeing, especially if they choose the download option in the player. Once the Visbit-encoded video is downloaded to the headset to play locally offline, the foveated rendering (their Visbit View Optimized Streaming method) totally rocks and latency is unnoticeable, which is absolutely perfect for what I need.
If you'd like more information about how Visbit is optimized for the various headsets it supports, here's another great video from Hugh Hou and CY Zhou (it's actually part 1 of the video referenced above). And if you want to read more about how to use the Visbit resolution benchmarking video toolkit in your own workflow, a very useful blog post about that is here.
My own personal proof (see for yourself).
I recently completed a new 360 video about aerial dance and it's being featured on Visbit's demo app. In my blog post about the piece I include a link to the 4K video that can be side-loaded onto the Oculus Go, so if you want to compare the 4K version to the 6K version on Visbit all you have to do is download the video, put it on the Go and then A/B the two versions. There is a clear difference.
There must be some alternatives, right?
Well, yes and no. The only foveated rendering options out there now (that I'm aware of) are:
John Carmack's 5K solution. I've read the paper about how he's implemented foveated rendering and watched Henry. The big takeaways are that he's compromising more (by not doing any high resolution encoding of the zenith and nadir areas) and the highest resolution this method uses is about 5.2k (which may not be optimal for down-sampling to the arbitrary position of the user's head at any given time). And of course there's the bigger issue that any videos encoded in this way cannot be streamed into the Go and must be side-loaded.
Insta360's upcoming Crystal View. Not available for actual testing yet (only sample files), but this looks to me like they've also implemented some version of Carmack's foveated rendering solution. I can't comment on quality until I can actually use it, but of course it suffers from the same problem of how to distribute files to collaborators and customers without forcing them to download files and manage various versions of those files on their devices.
Pixvana. This is the only other company creating 360° video tools in the same space as Visbit. I don't have any direct experience with their system but from what I've read on the Facebook VR Professionals forum, Pixvana doesn't have a fully foveated rendering solution as comprehensive as what Visbit is offering. I've watched a webinar recording with their founder Sean Safreed and he does mention that they're using Carmack's method of encoding, which is quite different from Visbit's approach. The cost of using Pixvana is similarly in the professional realm but Pixvana doesn't offer anything like Visbit's affordable "Creator" ($69/month) plan for individuals like me to get started with their system. I'm typically only working with a few collaborators at a time and the Creator pricing is clearly within my budget reach.
An important piece of an evolving 360° video infrastructure.
I originally discovered Visbit when Elaine Lu (a co-founder and COO) contacted me earlier this year regarding the 360° video I posted about alternative process photographer Lindsey Ross. She asked if they could feature the video in their demo app and when I saw the result in my Oculus Go it was definitely a case of "seeing is believing." And, as mentioned above, I recently finished a new 360° video featuring the premiere of an aerial dance performance and that was also selected to be featured in the Visbit demo app. See for yourself... the quality is outstanding. (Once you've downloaded , also download the video locally to your headset and view it that way for the absolute best quality.) Their interest in my work prompted me to get in touch and now I’m using the Visibit system in evaluation mode. The service is not inexpensive, but if you’re a serious professional this system makes a tremendous amount of sense. It’s important to support the nascent infrastructure that’s evolving in the 360° video space because if we don’t actually use the tools that these creative people invent and deploy for us, they’re going to have to move on to solving technological problems outside of the 360° space, which would be unfortunate.
Image quality is stunning, audio plays correctly and my collaborators and customers have a pain-free method to view my videos with a simple push of a button. Visbit is a godsend and solves one of my gnarliest problems.
We're getting closer.
My personal dream (which I think is shared by many of us) is that soon we'll have a solution for distributing 360° videos... kind of like a "Vimeo for 360." This would be a service (that I'd be happy to pay for) which would host videos and either stream or download them locally to any user's headset in the highest quality possible. I'm hoping that the equation that considers costs of bandwidth/encoding balances in such a way so a reasonable business model arises that works for both the distribution platforms and content creators. In the meantime some of that is here right now. I've been working in technology and imaging for >35 years and have seen enough to understand that all this is coming. Patience can be described as a way of not having everything you want in the present but being happy to hang in there and watch it all develop!