Launch HN: Vimmerse (YC W22) – Platform and SDKs to create and play 3D video
Will there be an Emacsmerse compatibility mode? ;)<p>But more seriously (but still on the spirit of Emacs), how will your 3D video player be scriptable and extensible at runtime?<p>I have some positive experience with extensible 3D panoramic stereo videos players, which I&#x27;d love to share with you:<p>I developed a system for JauntVR that enables cross-platform scripting of Unity3D applications in JavaScript, for Jaunt&#x27;s 3D panoramic video player. It also greatly eases and accelerates development and debugging, which is otherwise extremely slow and tedious with Unity.<p>Unfortunately JauntVR ran out of money for that project and pivoted to other stuff before publishing it, but Arthur van Hoff generously let me open source the general purpose Unity&#x2F;JavaScript scripting system I&#x27;d developed for it, which I call &quot;UnityJS&quot;.<p>It&#x27;s useful for all kinds of other things beyond scripting 3D videos, and it works quite nicely not just on mobile devices but also with WebGL, but scripting interactive 3D video was the problem I originally created it to solve.<p>I&#x27;ve used for several applications and integrated it with several libraries, including financial data visualization on WebGL, TextMeshPro in Unity, ARCore on Android, and ARKit on iOS.<p>UnityJS also makes developing and debugging scriptable Unity apps much easier and quicker (by orders of magnitude) because you can use the standard JavaScript debuggers (even on mobile devices) and just restart the app to reload new code in seconds, instead of waiting the 10-60 minutes it takes to totally rebuild the app in Unity.<p>By using the same standard built-in JavaScript engine of the web browser on each platform, you can implement cross-platform interfaces in JavaScript instead of re-tooling the interface for each platform in different languages, and incorporate standard up-to-date off-the-shelf JavaScript libraries.<p>I believe the 3D video player should be much more like an extensible scriptable browser (and like Emacs and NeWS!), instead of just a dumb fixed-function play&#x2F;pause&#x2F;rewind&#x2F;fast-forward video player (like piping a text file through &quot;more&quot; on a VT100).<p>So you can download dynamic content and scripts that implement custom interfaces for 3D videos, like interactive titles and credits, custom menus, theater lobbies, even networked multi player environments for watching 3D videos together.<p>It&#x27;s great fun to overlay live 3D graphics, text, physics simulation, interactive kinetic typography, particle systems, etc, into 3D videos.<p>It&#x27;s extremely useful for selecting and navigating content, displaying interactive titles and credits, or anything else you might imagine, like games and applications embedded in and orchestrating 3D video.<p>And of course you can publish new content, hot updates, and extend the experience by downloading new code and data over the internet, instead of pushing out a new version of the app through the app store every time you want to improve the experience or publish a new video or series.<p>Some obvious and useful application for 3D videos is designing compelling opening and closing credits, captions, subtitles, pop-up notifications, pie menus, player controls, navigation, bookmarking, tagging, and any other kinds of user interfaces, that can adapt to the content and environment in whatever direction you&#x27;re currently looking, so you don&#x27;t miss them and they&#x27;re not unusable, just because you were looking in the wrong direction at the wrong time.<p>3D videos suffer practically and creatively from hard-coded static title sequences and credits, because you might not be looking in the right direction at the right time to see them. It&#x27;s like pointing a movie camera at a stage to film a play, instead of inventing a whole new expressive cinematic language appropriate for the new medium. And I&#x27;m not just talking 3D scrolling Star Wars credits!<p>Of course you could just build a dumb non-extensible 3D video player with just one hard-coded style of menus and scrolling credits, or burn the title sequence and credits into the 3D video itself, but that would be static, boring, and inflexible, because the title sequence itself is an important thematic art form, which should ideally be designed for each different video, collection or channel.<p>The point is to enable the infinite creative possibilities of interactive titles and credits that respond to the viewer&#x27;s attention and interest: scrolling back and forth, revealing more information about the names you focus on in the credits, and all kinds of other cool innovations that run-time scripting, dynamic downloadable content, and interactive 3D graphics enable.<p>Saul Bass realized that 2D movie title sequences could be so much more than simply scrolling a list of names up the screen, and he designed the groundbreaking opening credits to movies by Hitchcock, Scorsese, Kubrick, and Preminger. Imagine how intense the interactive panoramic opening titles of a 3D film like Psycho or North by Northwest could be, to set the mood for the rest of the movie!<p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Saul_Bass" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Saul_Bass</a><p>&gt;During his 40-year career, Bass worked for some of Hollywood&#x27;s most prominent filmmakers, including Alfred Hitchcock, Otto Preminger, Billy Wilder, Stanley Kubrick and Martin Scorsese. Among his best known title sequences are the animated paper cut-out of a heroin addict&#x27;s arm for Preminger&#x27;s The Man with the Golden Arm, the credits racing up and down what eventually becomes a high-angle shot of a skyscraper in Hitchcock&#x27;s North by Northwest, and the disjointed text that races together and apart in Psycho.<p>Psycho&#x27;s opening credits<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=hwq1XHtJEHw" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=hwq1XHtJEHw</a><p>Saul Bass: North by Northwest (1959) title sequence<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=1ON67uYwGaw" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=1ON67uYwGaw</a><p>The Art of Movie Title Design | Saul Bass and Beyond<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=Q_Mo0MqICXI" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=Q_Mo0MqICXI</a><p>Saul Bass- Style is Substance<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=frWLpyI3lXY" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=frWLpyI3lXY</a><p>&gt;&quot;I have felt for some time that the audience involvement with the film should really begin with the first frame. You have to remember that until then, titles tended to be just some dull credits, mostly ignored, or used for popcorn time. So there seems to be a real opportunity to use titles in a new way. To actually create a climate for the story that was about to unfold.&quot; -Saul Bass<p>UnityJS is useful for other stuff too: Here&#x27;s a great example of an interactive WebGL based business model and structured financial data visualization system that I developed with UnityJS for Jen van der Meer of Reason Street.<p>The lion&#x27;s share of the code is written in JavaScript, plus a bunch of generic and bespoke modular Unity components and prefabs that JavaScript plugs together and orchestrates. On top of Unity&#x27;s 3D graphics, it uses the canvas 2D API and D3 to draw the user interface, info overlay panels, and translucent pie menus with live tracking feedback, all written in JavaScript so it&#x27;s easy to iteratively develop and debug!<p>It&#x27;s totally data driven by JSON data and CSV spreadsheets. So Jen can just enter fresh data into google sheets, turn the crank, and play around with the interactive visualizations within seconds. And I can just as quickly and easily modify the code or data, then quickly run and debug the new version!<p>Amazon FY2018 Visualization:<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=J8uOjelouUM" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=J8uOjelouUM</a><p>&gt;&quot;I created this tool as a way to kind of tell the story of how companies shift from e-commerce, and build these combinatorial, complex, technical system type business models. It&#x27;s a little bit of an obsession of mine. If you&#x27;re curious about this project you can find me in the links below. I&#x27;ll be collaborating with my long time collaborators here to see how complex companies and ecosystems [work], and help us all better understand how to think about these technology business models, and how they so substantially shape our world.&quot; -Jen van der Meer<p>Apple Services Pivot:<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=TR_4w7OCW4Y" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=TR_4w7OCW4Y</a><p>Facebook FY 2018 Financials Visualization:<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=cMQgjmj_mnQ" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=cMQgjmj_mnQ</a><p>Here are some link to some doc, notes, and posts about UnityJS I&#x27;ve written. And if you&#x27;re interested in using it, I have a more recent, modular, up-to-date version that I&#x27;m refactoring to use the Unity package system instead of git sub modules and symbolic links, and it also has some nice improvements for WebGL, and much better JSON.net&lt;=&gt;Unity integration:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;SimHacker&#x2F;UnityJS&#x2F;blob&#x2F;master&#x2F;doc&#x2F;Anatomy.txt" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;SimHacker&#x2F;UnityJS&#x2F;blob&#x2F;master&#x2F;doc&#x2F;Anatomy...</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;SimHacker&#x2F;UnityJS&#x2F;blob&#x2F;master&#x2F;notes&#x2F;talk.txt" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;SimHacker&#x2F;UnityJS&#x2F;blob&#x2F;master&#x2F;notes&#x2F;talk....</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;SimHacker&#x2F;UnityJS&#x2F;blob&#x2F;master&#x2F;notes&#x2F;unity-js-integrations.txt" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;SimHacker&#x2F;UnityJS&#x2F;blob&#x2F;master&#x2F;notes&#x2F;unity...</a><p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=21932984" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=21932984</a><p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=22689008" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=22689008</a><p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=22691004" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=22691004</a>