New research is giving computers the power to transform prose text into 3D worlds.
NOTE: This article will explain how text-to-scene conversion works and then go on to explore some of its applied possibilities. If you already have a good grasp of the tech or you’re just not here for that, skip to the bottom of the article for our predictions on how text-to-scene could change the way people see sex.
VR is hard work
As more virtual reality (VR) technologies are being developed, the demand for virtual worlds is increasing. Generating virtual worlds is not easy. Currently, they’re constructed through the tedious mouse-based process of manual 3D animation. This means literally thousands of “screen-hours” spent by highly trained professionals piecing these worlds together pixels at a time.
This method is not only tedious but very expensive. It means that only the most mainstream worlds get made because the massive investment required to commission them can only be justified by the safest, most broadly applicable ideas. In this creative climate, it’s very difficult to make original and exciting worlds into (virtual) reality.
Utilizing existing creative output
“This paper describes a strategy for automatically converting fiction text into 3D animations,” begins one of many ambitious research papers written by Shaun Bangay, Kevin Glass, and Bruce Alcock.
Bangay is a senior lecturer in Virtual Reality and Games at Deakin University in Melbourne. During the last decade, he’s been investigating and developing the possibility of a text-to-scene conversion system. This system aims to take the detailed worlds described in fictional works (novels) and use them as the building blocks for virtual worlds.
How computers work (in a nutshell)
As you probably know, computers crunch binary code, a series of 0s and 1s, switches that are either on or off, nothing else. It’s this enormous configuration of 0s and 1s, ‘ON’ and ‘OFF,’ across millions of terminals (semiconductors) that power all computational processes.
A program (or an app) is something that interprets and generates these 0s and 1s, using different programming languages to execute desired operations. Each of the many programming languages is a coding system that interprets 0s and 1s differently, using highly specific grammatical alphanumeric systems.
How text-to-scene conversion works
Text-to-scene conversion can be imagined like a language bridge between human language (fictional prose) and computer language (binary code). The conversion tool’s primary aim is to access the huge creative base of detailed imaginary worlds already existent in written fiction for VR.
The system is a text recognition system. It isolates words in a text into different types. These types include:
- Character/Avatar: Identification/description of a character
- Setting: Identification/description of the setting
- Objects: Identification/description of objects in a scene
- Spatial relations: Identification/description of the layout of characters and objects in a setting
- Movements: Identification/description of things that change position
- Scene change: Identification/description of transitions into new scenes
Each of these word types correspond with a parallel database of 3D objects, colours, textures, structures, and motions. These objects and motions can be automatically modified and combined to generate new objects. This gives the final 3D presentation far more variation and subtlety than a one-to-one system where words always correspond with a single 3D form.
You can look at an info-graphic describing text-to-scene conversion in one of Bangay’s papers here.
Text-to-scene conversion and human input
Human language is amazingly complex. A computer’s main problem understanding human language comes from that fact that it’s not universally standardized. Even within the English language, speech patterns and styles change with geography and time, and according to the age and social affiliation of the speaker. Some words have completely different meanings or simply don’t exist between two different places and/or times.
Humans are geniuses of contextual understanding. When we encounter words and grammatical forms we haven’t seen before, we can usually generate a partial understanding and continue to communicate positively with the new form. To compensate for this difficulty, the text-to-scene conversion system requires input text to be prepared and annotated before and after its conversion to a 3D representation.
Bangay and fellow researcher Kevin Glass write: “Fiction writing tends assume much implicit knowledge, and therefore any details not explicitly mentioned are not automatically annotated. As a result, there is an opportunity for manual intervention in the process during which human creativity may be used to insert further annotations into a scene.”
Human input is required at many levels throughout the multi-staged 3D construction process. Descriptors need to be linked to characters, the duration of events needs to be set and/or modified, scene transition and simultaneous non-visual events need to be integrated somehow.
What this means, for sex
At first glance, the realization of text-to-scene conversion means that books can be made visual and auditory, so we can watch instead of read them. This is true but falls short of this technology’s real potential.
Text-to-scene conversion can use books as first drafts for other worlds, cutting out thousands of hours of expensive animating time. This means more virtual worlds and a democratization of world-making ability—almost anyone with a computer can make one!
With development in haptic technologies, people could potentially insert themselves as avatars into a scene from one of their favourite novels. Before entering the text-based world, the user would annotate the text to allow for themselves as an extra character and set the parameters for their interactions before suiting up, seeing, hearing and feeling the scene as a virtual reality.
Some of classical literature’s most fiery relationships were described, to say the least, sparingly. Books create moods that allow the reader to interpret and create the best images for them. This technology could allow individuals to annotate classical texts and literally show others exactly how they imagined Wuthering Heights’ Catherine and Heathcliff, Pride and Prejudice’s Elizabeth and Mr. Darcy, or even Lolita’s Humbert and Dolores.
Fifty Shades of Grey, a more modern novel, has already been given to filmic adaptation, but if reviews of director Sam Taylor-Johnson’s efforts were anything to go by, the film’s imagining fell short of individual reader’s ideas about how E.L. James’ novel should have been.
Literary erotica, a genre with huge reader participation, is fertile ground for a text extension technology. Thousands of people contribute to personal and public blogs, churning out thousands of erotic fan fiction stories every day. Fifty Shades of Grey was originally written as fan fiction, designed to re-imagine the love affair between the Twilight saga’s protagonists, Bella and Edward.
People have written their own fantasies using existing characters from Harry Potter to Star Wars, and sometimes characters from different sagas meet. One Direction erotica has become a genre unto itself with tens of thousands of rated, erotic stories available online.
Show, don’t tell
“Show, don’t tell” is an often-mentioned axiom of fiction writing. It basically means that to write good fiction, the writer must display the scene so it is there in front of the reader instead of explaining it, or revealing it with secondary information.
Perhaps ironically, this rule may never have been better satisfied than with the application of what text-to-scene conversion could offer. It massively increases the potential for sharing imagination and constructing tangible realities that others can experience, refine and even participate in.
Sex, while a constant topic of conversation, has never been something you can really explain. It has a certain ineffability that demands experience and participation for proper understanding. While text-to-scene conversion is by no means the technology that will make virtual sex equal to the “real thing,” it’s certainly a huge leap in that direction
In what literary sex scene would choose to participate?