How To Invent Cinema


Thomas Wallner talks about cinematic Virtual Reality.

– translated from the Polish original “JAK WYNALEŹĆ KINO

Illustrations courtesy Édith Carron – Leipzig 2014

I see your company as futuristic, already being in the future. Tell us what is there? What do you think is the future of film?

I wonder if you will be the one who is going to make me eat my words 10 years from now – here is what you said, here is really what happened.

And you will have to actually eat your shoe, just like Herzog did.

Or both shoes. I will eat both of my shoes, if virtual reality doesn’t play a major role in ten years in the realm of cinema. I think the way to think about it is that regular cinema is not going away, it is a very strong art form. Theatre didn’t go away when the cinema came, it maybe just diminished in significance. I think VR opens the possibility of a new kind of cinematic experience, and if it develops in the right way into something usable I do think it is going to play an important role.

So how does VR work? You put a let’s say Oculus headset on and then what happens?

You are looking at a screen, but as your head turns, the screen shifts, so the computer knows that you are changing your perspective, and it tricks you by showing you new images for the direction you are looking in, so that it feels like is that you are looking around an actual space as if you were there.

What’s the VR experience like?

We can describe what virtual reality looks like, but we cannot really convey what it feels like. It is something you have to experience. It is an experience that didn’t exist before. I also can’t show you a film or a photograph that conveys the experience of virtual reality, because the experience can’t be depicted in another medium because it is unique to itself. One way to explain this is I could take a photograph of you right now during this conversation, but your sense of being here at the table talking to me is not describable in photographic terms. If you shoot a film of our conversation it’s a recording of this event but it still does not feel like being there if you watch it. If you forget about the technical aspect of virtual reality for a moment the best way to describe it is that it would feel just like sitting here with me having conversation. It is being transported somewhere else, into a moment. That’s the best way to describe it.

Does VR being the future of storytelling mean in the future people will be having headsets at home, just like they used to have TV sets?

Right now the hardware that we are using, especially these early viewers, these big black boxes that everyone is strapping to their faces are very bulky. Let’s face it, they make you look like an idiot. Right now these systems are made of readily available components like mobile phone screens and cheap plastic lenses. We are dealing with really early prototypes, that people are putting together to create that VR experience. They are not streamlined in any way. I think down the line what we will be looking at will be lighter, smaller, have more resolution per eye, it will be much more comfortable, and there won’t be big wires. Already the mobile devices don’t have the wires, so I think it will be a much more elegant experience in the future. I also think there will be front facing cameras with light and depth sensors that allow for detecting the movement of your hands in order to interact with virtual objects and interfaces.


What will be the content, and what would be the experience, and where would they use it?

That is a big question. When I visited the people at Oculus, the makers of the Rift headset, a couple of weeks ago, one of them said: whatever we think VR is, it is certainly not what it is going to be. And that is an acknowledgement that we are in an early phase of experimentation that will lead us to discover what VR was meant to be.  So it is hard to know exactly what that is going to be right now. So all we can do is experiment, hopefully systematically and through success and failure, sometimes perhaps by accident discover the aesthetics to tell a story in this medium. So I can’t tell you what it is going to be, we’re experimenting and trying to figure this out. In that way we are like the first filmmakers, who were shooting the first pieces of film simply because they had a camera that was able to record reality. It then took many decades for them to determine how to turn it into a visual language we call cinema.

I don’t think it will take that long with VR, because we already have all his experience with film language, with gaming conventions, so I think we can transfer, adapt and build on a lot of this knowledge. And the exchange of knowledge nowadays between groups of people is so much more rapid and information travels faster, so innovation happens faster. So I don’t think it will take as long to arrive to these language conventions as it did when film was evolving as a medium. Also the technological means to achieve this and have become cheap enough for hundreds of thousands of people to experiment with this as we speak.

Do traditional filmmakers engage in the medium? You met with Werner Herzog to present to him the VR technology. How did he react?

I was travelling down to the first ever VR conference in Los Angeles, organised by Oculus. I was asked by a Canadian producer to meet Werner Herzog to discuss a project that he is embarking on called Into the Inferno, a documentary and IMAX movie that explores the impact of volcanic eruptions on the course of human history. The Canadian producer on the project has the intention to do something interactive with Werner Herzog on this, which is why I was brought into the project. Whether this happens or in what form remains to be seen.

I met Werner Herzog at his house, and we spent a nice afternoon talking about various things. And I brought the Oculus Rift along and I let Mr Herzog and his wife Lena try it out. We then had a discussion about the significance of Virtual Reality, and of the challenges that it poses. And for me it was a very interesting discussion, I am a filmmaker, I know all about story, but I am also looking for solutions.

Mr. and Mrs. Herzog didn’t find solutions this afternoon, how could they – we are all looking for them basically but it was a very good beginning dialogue. Later I brought Werner and Lena to the Oculus Rift conference, and we then met with other people working in the realm of VR space, people who are really at the forefront of this medium [Chris Milk, Felix and Paul, Kite and Lightening]. It’s a funny memory for me, I always admired Werner Herzog very much, and it felt the meeting at Oculus had a sort of historical dimension to it.  To see one of the old great masters of cinema around the table with a younger generation that is trying to tackle the technological inevitability that the Rift represents head on by finding a way to turn VR into an art form or a new way to tell stories was a very exciting discussion to be part of. And after this Lena and Werner wrote a 2-page paper that summarized everything we discussed and sent it to all of us and I thought it was a very good distillation and insightful analysis of the challenges of defining ways to tell a story in this medium.

    FullSizeRender         Photo: Courtesy Lena Herzog

What are these challenges?

We come across these challenges very quickly as a filmmaker. One of this challenges – Herzog recognised it, we all recognise that – is that when you are shooting the world spherically you are shooting it from the inside out. So you are at the center of the scene, and the world is around you. He and Lena, describe it very astutely as there being only an outside and no inside. In a static film you can project your feelings and desires on the characters – let’s say when two characters interact,  but if you are so deeply embedded, you don’t know what you are, you are a observer in the middle of a spherical universe, it is very difficult to follow that mode of storytelling. And of course traditional film is defined by the juxtaposition of images, the collision of images to create a feeling and a meaning that often isn’t even contained in the images themselves. In cinema we create that meaning in our head.

We all know the Kuleshov experiment of a man looking at a child, at a women, and at a dead child, and if you connect a shot of him gazing into the camera into three different variations of film you will say “the man is so sad, because the child is dead” or he is looking lustfully at the women or look how hungry he looks when he looks at his soup, we are creating that meaning in our head. This is what they call magical symbolic thinking. Which we apply to film and actually to every other narrative medium until Virtual Reality. Because we perceive reality in fragments that we put together. I think film follows that cognitive process of the brain, to take fragments of reality and assemble them. So film is actually a metaphor of reality in fragments that we intellectually assemble. And that is the great art of cinema. We are using a natural way in which the brain processes reality – by making it into a story. The film medium has been made to speak that language. Virtual reality at first glance seems to work completely differently. Virtual reality hijacks our senses and creates a new reality around us. There are no cuts, it just is, and as wonderful as this is, it is also problematic for telling stories, because traditional story telling, especially in film bends time and space to get to the essential part of a story to transport its intended meaning, whereas in virtual reality there is a persistent aspect to it where time is time and space is space. If you were to rapidly cut it, it would feel like a schizophrenic episode, when you are in one moment here, and the next moment somewhere else. So the question is – do we have to slow it down a little bit? Do we have to have approaches that are similar to the theatre, where the scenes are unfolding around us, it is very hard to know. We are at this beginning where we really need to experiment to find out. Thinking and talking about it is not enough. We need to play to see how it feels. I think we may be looking for a form that works in a stream of consciousness kind of way, where the images morph and blend in front of us like a dream. This would actually utilize both the temporal and special qualities unique to VR. But that has to be experimented with and we may be overthinking it.


The people who are game developers, they have been facing this challenge in the narratives of the games.

I think people see games as the obvious use of Virtual Reality. It is kind of true. I think games developers are facing the same challenges as people who want to import cinema into VR.

For instance forward movement, rapid forward movement makes people instantly nauseous. Imagine all these action games without any forward movement, how would you ever do that? And you would not build a game where 80% of the people feel like vomiting after 3 minutes of game play. Also games developers are facing these challenges of finding new ways to make games and they will have to resort to a new language and conventions of game play as well. And by the way, narrative has never been that important in action oriented games. But I think the inventors of Virtual Reality that were strongly focused on game have consistently underestimated the importance of VR as a foundation of a new kind of cinema. I actually think it may become a more popular use of VR than gaming, because it asks less of the viewer. Games are lot of work but the transportational power of VR that overwhelms people upon contact that leads to a sense of wonder and awe is immediately instilled in even the most primitive cinematic VR experiences. That is very promising.  We now have to find the kind of story and experience that will uphold that sense of awe and wonder long after the fascination with this medium itself has inevitably worn off. Right now is kind of a honeymoon phase were the fascination with the experience of the medium is enough to engage audiences. That won’t last.

You too had to face some of these challenges while working on your first VR project, The Polar sea 360’. Why did you choose this medium in the first place?

The idea for the Polar Sea originated at Arte Germany. The CEO of Arte Germany, Wolfgang Bergmann wanted a 10 part TV series that travels through the mythical Northwest Passage.

Right from the beginning he wanted a concept that was more than just a TV series; he wanted to expand the series into other media. At the time I became aware of the emerging technology of shooting reality in 360 degrees. Mr. Bergman’s other wish was to create something that would give the audience a visceral experience of the Arctic. It wasn’t about more information or knowledge, he wanted to touch people so they would care about climate change, as a result of the experience, not because it was the right thing or the urgent thing to do.

We felt that using immersive film would be a great way to achieve this. When we started the project we wanted to create 360 videos where the viewers could look around the Arctic for themselves. This was intended as a screen based experience for the web. As the project progressed over a period of 1,5 years, VR devices like the Oculus Rift became available to developers like us. So we took the spherical footage that we shot in the Arctic and tested it inside the Oculus Rift. And that was a very special moment when I realised that this was very powerful new medium, that could literally transport you somewhere else. For the first time in the history of media, we have a medium that breaks the fourth wall, alleviating the distance between the observer and the observed, and from then on we decided to add a VR component to our project, so the audience of the Polar Sea could literally feel like they are there, in the Arctic, which in a wonderful way fulfilled the original intension of what Mr Bergmann wanted to do.

We-Need-Story-for-webWhat about filming, how did you shoot it?

The way we did it is that we took not one camera, but six or seven cameras.  You have to imagine that the lenses are pointed outwards from a central point, the cameras are equally distributed in a ball-like fashion facing outward. So each camera shoots a fragment of the reality, and in post production you assemble, you stitch together the fragments to make a perfect sphere. That is how you do it.

At the end as the viewer you get this object, which is the Oculus, and suddenly you can just go the Arctic. Suddenly with your sight, with you body, you can be somebody else, in a different universe, and it is almost like getting supernatural powers. Do you see it that way?

There is wish for us to be able to create another reality and inhabit it. I believe every storyteller from the beginning of time wants to do that. Whether the story is being told around a campfire or in a theatre, or in a film, storytellers want to transport us to another world of their making.  And they all do it using various mediums each with its own imperfections that create a kind of distancing effect we overcome by suspending our disbelief. So in theatre you are generally still aware of the fact that an audience is sitting there with you looking at the stage in front of you. In film you are aware of the screen to a degree.  In a book you are still aware that there are words on a page you have to weave together to ignite your imagination. And by the way, that is wonderfully immersive. In fact our suspension of disbelief that is at the heart of all traditional media enables us to forget the medium itself once the experience or story takes hold of us. But there is always a distance between you and the medium. You have to slip into a kind of story trance to let go and forget this. We’ve become quite good at this. In virtual reality you can now directly hijack the senses of the audience and that could be be a powerful new tool for a storytellers. We can create worlds and transport an audience right into their literal manifestation. That has never existed before.

When you are in a theatre, somebody from the public can stand up and disturb this experience, with books you have to turn the pages. In other media this interactive experience is interrupted. While here, I imagine you would have this experience, where you just immerse yourself. Some might argue that on a psychological  level this sounds scary and addictive.

We don’t know. Every time a new medium came about, there was somebody arguing this might be scary and addictive. When television really started to infiltrate our lives, there was a growing sense of alarm that we may become addicted to TV. Don Quixote is about the addiction to books and ideas. The internet is addictive. Every time something new comes, we project our fears into it. The other thing that comes up is people pre-assessing VR as a really lonely isolating experience when you put those VR headsets and noise-cancelling headphones on, arguing that traditional film and cinema are highly collective experiences in contrast.

Based on that argumentation there is this other really scary medium that we should stop immersing ourselves in right away, and that is reading. What we forget is that reading was once a collective medium when books were precious and shared by a person reading to a group before it migrated through technological innovation to become a solitary medium. Nobody worries about that anymore.

On the other hand, what I think it is nice, is that when you are in Virtual Reality you are actually shielded from all other media and the disruptions around you. So there is a focus and concentration that has become rare in our relationship with the media around us. Many times we are now watching a film on a tiny screen while reading our e-mail on the same screen. Right now the way VR works it completely isolates you and transports you into a moment or experience without distraction. Total concentration. That is actually amazing, I love that. When else would a storyteller get that kind of devotion from the audience?

Media is either communicative or transportational, and I think we have a desire to be taken somewhere else, and this medium does that for sure. Because it hijacks your senses, it really tricks us, it gives you a sense of presence of being somewhere else, which is not here. That is something very powerful and very seductive. And we have to figure out how to use this part of the medium to its maximum effect. When you have a new medium you have to figure out what are its strengths and how to utilise them. But I do believe that if we don’t manage, and I think we will manage, but if we don’t manage ultimately to make a medium meaningful, able to tell a story, it will not survive. Media ultimately has always been there to tell stories or to transport meaning of some kind.

So the idea would be to have this uninterrupted experience. Do you think you would be able to make it happen?

The real question is how long are sustainable experiences. Right now the headsets are not that comfortable, they are put on your head using elastic bands, it get hot in there, the lenses fog up. How long do you want to stay in a place like that? I don’t know. I think if in the future it becomes lighter and easier you might stay longer, but right now it is a bit uncomfortable. And honestly we don’t need to create long experiences yet, because we still have to learn how to create short engaging and meaningful experiences that are not just tech demos. You have to learn how to walk before you run.

Also Virtual reality is almost like real, but it is not reality. You make people think things are real, but they are not. They almost are.

I believe the power of VR will also be in experiences that are real, not computer-generated. When you see a character that is created by a computer programmer and a designer to look like or simulate human beings, every pore of our being knows that is not a real human being. The light that fell on the face of an actual human being that was captured by a camera has a different sense of authenticity about it. That is something that I call “the aura of the real”. And I think that speaks to us. And when artificial computer generated characters, become more and more like human beings, we start to feel a discomfort. There is a term for that, it is called the Uncanny Valley. Unheimlich as the German’s call it. The word is the negation of Heimlich or feeling at home. Uncanny. Because we know it is not human, which is why for instance in the design of robots engineers are going over to have non-humanoid android forms. If androids are human-like, but not human enough, it looks fake and tacky. And if they are too human, people actually develop a dislike even disgust for them. So I believe the power of VR will also be in experiences that are real, based in the photographic depiction of reality for that reason.

Did people always go to go in this direction, immersive reality?

When we look at the writings about early cinema, in the latter part of the 19th century, what happened is that people saw the Kinetoscope, they saw photography, they saw the first examples of moving pictures, and in their mind they immediately though “we can actually capture replicate the world in moving pictures”.

You have to remember, at that time there was no camera, there was no screen, there was an assembly of still photographs spinning a cylinder called a Kinetoscope. And of course people saw that an imagined what that medium would do in the future. They did not imagine a screen and a projector, which came a few decades later, because that was only a temporary manifestation of cinema utilizing the limited technical means of that time.

What they imagined was something like the Holodeck, 3D, 360, all around you. Something that would capture reality perfectly and took you there. They imagined VR cinema before the cinema we are used to was even invented, even before the first screens and projectors made their appearance and to that is really quite incredible!

French film theorist Andre Bazin said the early thinkers of cinema, before cinema was invented envisioned what he called Total Cinema. Cinema that was 360 and around you, that was a perfect copy of reality. When Bazin died in 1958, when cinema already had colour and 3D and sound, he said that cinema hasn’t been invented yet and if you expand on that line of thinking, that idea, yes we are now getting closer to inventing the cinema that was imagined at the dawn of cinema in the 19th century.


So do you think the cinema has been finally invented?

No, still not. VR is a first step towards the realisation of total cinema. And there other technologies that are in a nascent state that will play a huge role in the future. Lightfield photography, body scanning, artificial intelligence etc.  I think the ultimate dream is to have life that unfolds like a story around us, and no doubt approximations of this will be created in the future. I think these technologies are the expression of our ultimate desire to cheat death by capturing life in its entirety and our ultimate desire to be god-like, to create a universe that you can put our human actors into, that somebody else can experience as an extension of their existence. That would be Total Cinema. Perhaps we will get there if our collective hubris, that is reflected in these blasphemous ambitions doesn’t actually destroy us first.

Interview conducted by Anna Desponds – Amsterdam 2014