Our monthly news update this month covers more genAI matters, game-related items and WoWness in the form of a 2 hour + stunning orchestral performance celebrating 20 years of musical history. Check out the news and our discussion, and do leave comments too!
YouTube Version of This Episode
Show Notes & Links
A new AI-focussed filmmaking studio has launched, called Asteria. Its argument is that AI can change the way indies make and distribute content, providing new opportunities, and presumably new types of content – its very much the same point we made a few months back about mAIchinima. Link here.
The Silent Hill 2 Remake with a mod that turns it into a VR game, made by Praydog that gives the game more of a first person perspective. Demo video here –
Philip Rosedale has rejoined the team at Linden Labs’ Second Life as the CTO – roundtable discussion with the community in November here –
There’s also a great interview that James Wagner Au has done in two parts on his New World Notes blog here
Remarkable Life of Ibelin trailer here (feature image) –
World of Warcraft 20 years of music orchestral concert, video of performance here –
Steam Launcher update now with recording tools for machinimators. News item here.
Dragon Age: The Veilguard official launch trailer –
I was privileged to have been invited to Oberhausen International Short Film Festival (Germany, 26 April to 1 May) by Dr Lars Henrik Gass, Festival Director, Katharina Schroder, Theme Coordinator and the programme curators, Dmitry Frolov and Vladimir Nadein for the first programme of avant-garde and experimental machinima films at a major European film festival to take place. I was also delighted to see machinima work, created by Alice Bucknell, featured on the programme magazine’s front cover too! I was there, primarily, to participate in a panel discussion with Gemma Fantacci, one of the organisers of the Milan Machinima Film Festival, and two artist-curators from the US and Hong Kong respectively, Alice Bucknell and Ip Yuk-Yiu.
Programme
The programme, entitled Against Gravity, was themed to loosely represent a video-game/machinima creative experience, from Starting the Game, Holding the Controller, Crack the Code, Don’t Forget to Save, Opening the Map, Unlock the Real, Cosplay As…, and a retrospective of Phil Solomon’s work, called Interplay, a filmmaker whose turn to machinima in later life was the inspiration for the festival theme. As one might expect, the organizers dive into each sub-theme to tease out their narrative through a selection of works across a span of 27 years of machinima practice. Unfortunately, I didn’t arrive in time to see all the selections, but the ones I did see were very interesting.
Image: Programme Cover (link to Festival programme here)
Many of the films were ones I had not previously seen – experimental films tend to be distributed in a different way to traditional machinima, using mostly artistic channels, and so often make their way directly to independent film theatres and galleries as a consequence. This of course diverges from the traditional approach to machinima-making of community-shared content, through which debate and practices are openly discussed. What was interesting with the presentation was the mix of older work with more recent pieces, effectively positioning machinima in a narrative of avant-garde practice and demonstrating experimental applications by artists/filmmakers for an audience that clearly had interest in the work if little experience of machinima or games specifically. I got the distinct sense of the emergence of a new generation of machinima fans – both creators and audience – and also the sense there is demand for a new way to experience machinima on larger screens from this audience, especially with technological advancements that make quality of content higher resolution and approaches employed by artists accessible in the experimental tradition of film viewership.
It was also interesting that the foundation of machinima was considered relevant to the presentation of avant-garde works both because its origins lie within 3D real-time games engines and also because the use of game as a creative matrix for storytelling beyond gameplay conveys subversion, virtuality and transcendence – all themes that resonate well with the experimental arts movement. Drawing on Solomon, however, the rationale for including machinima was the transformation from analogue forms of film to digital, with all the nuances that brings – relating to the breadth of creative practices, methods of transferring work for exhibition, storage and also presentation to audiences. What was interesting in this is that many at the festival seem to have little conception of either the term machinima, the game communities from which it emanates or its impacts on popular online culture. Filmmakers selected appear to have forged their own practices, sometimes as multimedia artists, using games as found tools and environments. Some were clearly also a little reticent about being associated with its background and origins, the taint of the M.com years, and probably alongside that the many issues the community has collectively faced in relation to the recognition of originality, ownership and authorship of works created.
Of the themed selections I saw, in Open the Map, the idea that games whet the appetite for utopian experiences was demonstrated through the selection of two films. The first was a documentary about a group of queer teenagers living through the challenges of Covid by escaping into Minecraft to connect with each other in a virtual safe space they made their own. The film, Tracing Utopia (2021), was directed by Nick Tyson and Catarina de Sousa, and was based on their observations of working with the teenagers over an extended period of time. It highlighted well the essence of community and collaboration, something that has always been the heart of machinima. The second was Alice Bucknell’s three-channel installation converted for the festival to a single channel, called The Martian Word for World is Mother (2022). This film was made in Unreal Engine and showed three different visualisations of Mars (the Green, the Blue and the Red), each being a post-human perspective on the future colonisation of the planet, drawing inspiration from popular sci-fi tropes on the interplay between ecological, economic and political ideologies. These include Elon Musk (terraforming), Donna Harraway (Cyborg Manifesto), Ursula K Le Guin and Kim Stanley Robinson’s 2009 Mars trilogy, among others. It is Bucknell’s Blue Mars that is shown on the front cover on the Oberhausen Film Festival programme, illustrating how multinational organizations convert permafrost into water for sale back to Earth – something she presents as a speculative near future. Not exactly a vision of utopia per se but her Green Mars was an interesting approach to reflecting on how the technologies used to exploit Mars in her Red and Blue versions may help Mars bring back to life its former natural habitat for its own end. She includes a mystical (textual and aural) language that was created using Scottish Gaelic, Helene Smith’s invented Martian language and sounds of Arctic wind, created using a text-to-speech AI and the assistance of OpenAI’s GPT-3. A fascinating creative process and captivating to watch and listen to.
image: Tracing Utopia
In Unlock the Real, older films were used to position machinima through documentaries, for example Harun Farocki’s Parallel I (2014) showed how games have advanced in their representation of assets such as trees, clouds, water and fire from the earliest machine dashes in the 1980s, to squares of the early 1990s, to more modern mesh representations such as in GTA in the 2000s. That reflective process of course, with recent developments in the representation of fluid, skin and movement dynamics integrated now into so many tools that creators use, whilst incomplete in Farocki’s work perfectly illustrated how games have focussed on advancing realism. This was then followed by works that demonstrated aspects of even greater realism, from shape and form, to speed of movement (Benoit Paillé’s Hyper Timelapse GTAV, see below), to representation of human experiences and the interplay between virtual and real, and even the way that in-game corruptions of brands really don’t hide what they represent any more (Jacky Connolly’s Decent into Hell, 2021). Indeed, Connolly’s clever interweaving of photographs and real film into the GTAV landscape was quite a trippy experience. It is not often one sees that trajectory so clearly but Vladimir and Dmitry’s curation made it easy for less experienced machinima followers to make the connections between virtual game worlds, film and real-life experiences.
Benoit Paillé’s HYPER TIMELAPSE GTAV (CROSSROAD OF REALITIES) (available on Les Nuits Photo Festival Vimeo channel, released 30 Nov 2015) –
The Phil Solomon retrospective, themed Interplay, was equally fascinating, with introductory comments by Ip Yuk-Yiu and Lynne Sachs who both knew him personally. Solomon was a key influencer in the US avant-garde film scene, and one of his early works had previously won an award at the Oberhausen Film Festival (Remains to be Seen, 1990 – a clip of which you can see on this Vimeo channel). Vladimir and Dmitry selected two of his early photochemical 16mm films: The Secret Garden (1988) and Twilight Psalm II: ‘Walking Distance’ (1999). These had a mesmerising depth to them apparently achieved by applying an emulsion to each frame and then transferring it using an optical reader. They were juxtaposed with the subsequent selection of Solomon’s tribute following the passing of his lifelong friend, Mark LaPore, in a trilogy of machinima films made using GTA (San Andreas). The trilogy was called In Memoriam: Rehearsals for Retirement (2007), Last Days in a Lonely Place (2007) and Still Raining, Still Dreaming (2008). Interestingly, Yuk-Yiu commented that he never fully appreciated Solomon’s machinima work when it was released. This was primarily because he could not see a connection to his previous work yet the interplay was evidently between chemistry and code (analogue and digital) techniques as well as presence/absence represented by the virtual world. However, the rather melancholic scenes in the machinimas, which seemed to somehow represent Solomon’s search for his friend through the glitches in the game world, may also have been connected through another film that Solomon made with LaPore in GTA, released just a few days before he died, called Crossroad (2005).
image: Vladimir Nadein and Dmitry Frolov introduce Interplay with Lynne Sachs (right) and Ip Yuk-Yiu (left)
Panel Discussion
I really enjoyed the opportunity to participate in the panel discussion as well as the various follow-up chats with others visiting the event. On the panel, each of us had very different experiences and perspectives of machinima, and we were asked some great questions about how we came to machinima, how we see it evolving and what its future will be. A question that emerged was at what point is the term machinima no longer relevant. From an avant-garde perspective, some questioned whether machinima’s history got in the way of framing their work. Well, of course, many of us long-timers have had those discussions over the years but it was interesting to hear others discuss this too. I really liked how the critical reflection suggested that it is its adaptability in reflecting latest technological and game advancements as ‘machine cinema’ that keep it apart from other creative practices – an observation I like to think the pioneering Hugh Hancock would have been supportive of.
Machinima is also clearly a good fit with the avant-garde scene, which of course is well reflected in Matteo Bittanti et al’s VRAL Patreon project, where you can find monthly film selections and interviews with creators. Matteo is also one of the founders of the Milan Machinima Film Festival, along with Gemma Fantacci, a student of game-related counter-culture, whom I had the pleasure of meeting in person for the first time in Oberhausen (although earlier this year we also chatted for the Digra Italia talks series). Indeed, a number of the directors included in the machinima theme at Oberhausen were ones the Milan team have featured, as indeed have we on the CM podcast.
Another interesting audience question was also about the nature of 3D and its continual re-emergence in film, and how it differs in games. It was clear that with a primarily film-based audience, some struggled to appreciate the nature of 3D and realtime concepts that are so well understood in game worlds. Of course, anything rendered for a 2D screen immediately makes it difficult to imagine depth dimensions, or indeed the methods used to capture and edit scenes, for example using mod tools within 3D environments. Part of the reason is the different uses of the term 3D: in-game experiences are a 2D illusion of a 3D representation rendered through a screen. 3D films in contrast are a fixed view perspective generated by overlaying stereoscopic views. I’m sure they exist but, thinking about this, I don’t recall ever having seen a 3D stereoscopic machinima. Nonetheless, the nature of machinima remains a challenging aspect to communicate and convey – as indeed do the differences in the experience one has when viewing the work in a publicly shared environment such as a cinema, gallery or arcade compared to desk-based screens, handheld or even hyper-personal VR screens. For example, most machinimas we review on the CM podcast have never been intended for consumption on the ‘big’ screen and it certainly isn’t made using large format screens. This is something I recall thinking about at great length for the 2007 European Machinima Festival as well as the various showcases of works I’ve made over the intervening years, not least because detail is emphasized in ways that is never really experienced in a 3D game environment. And even though many avant-garde works are intended for larger format screens, this aspect does not seem to be a particular focus of directors. Thus, the 3D/2D transformation highlights another important yet completely unexplored and unappreciated aspect of machinima as avant-garde or experimental film, perhaps related to the closed distribution methods and consumption experience design strategies used as previously highlighted.
In sum, Oberhausen International Short Film Festival was a fascinating in-depth review of different avant-garde perspectives on machinima. It was a real pleasure to have time to talk to some of the participants in the programme who made the journey from across the world to attend. I was especially thrilled to see the baton for the development of a new appreciative community being picked up by Dmitry and Vladimir – the huge effort they had made in devising the programme, engaging with creators and connecting the dots across the generations of machinima and filmmaking traditions was outstanding, and evidenced in most of the theme sessions selling out. I was really pleased to hear they are potentially going to be doing more of this in future and I’m certainly looking forward to catching up with them soon to hear their reflections and future plans too.
And finally, something I haven’t experienced since I was a teenager was the beautiful plush setting of the cinema in which the programme took place, complete with red velvet curtains and projectors (when required) at the Lichtburg (meaning ‘fortress of light’) Filmpalast Gloria auditorium (see above). Oberhausen is the oldest German short film festival, founded in 1954, and one of the most significant for the development of production conditions in Germany. In 1962, the Oberhausen Manifesto declaration by 26 filmmakers at the eighth festival is attributed with having created the basis for the success of New German Cinema worldwide. I therefore thoroughly recommend finding the time to visit and support a future festival.
Summary of Themes and Films
A listing of the films shown for each theme is as follows –
Start the Game
Everyday Daylight by Total Refusal (Austria), performed live with GTAV
Vladimir Nadein (b. 1993, Moscow) is a curator, artist and film producer based in Taipei, Taiwan. His works were presented at the solo exhibition Deep Play, VT Artsalon and Greater Taipei Biennale. He produced an award-winning film Detours, supported by Hubert Bals Fund, received the Eurimages Lab Project Award at Les Arcs Film Festival and was shown at Venice Critics’ Week, Viennale, Thessaloniki IFF, Berlin Critics’ Week, FICUNAM, Jeonju IFF, IndieLisboa, Beldocs, FILMADRID, Camden IFF, TFAI, Barbican Centre among others. In 2016, Nadein co-founded the Moscow International Experimental Film Festival and directed it for five editions. He curated special programmes and screenings for the 17th Venice Architecture Biennale, Hamburg Short Film Festival, Moscow International Biennale for Young Art, Garage Museum, University of California, Los Angeles and other venues. Nadein tutored at the Moscow School of New Cinema and is a member of the filmmaking duo together with Dina Karaman.
Dmitry Frolov (b. 1988, Kaliningrad) is an art and film curator and researcher based in Izmir, Turkey. He holds a BA in Сultural Studies from the Russian State University for the Humanities and an MA in Film Programming and Curating from Birkbeck, University of London. He has curated a variety of screenings, panels, performances and exhibitions dedicated to such artists as Maya Deren, Chris Marker, Tony Conrad, Vladimir Kobrin, Yoko Ono, Michael Snow, Annabel Nicholson, James Benning, Alain Cavalier, Aura Satz, Cao Fei, Ana Vaz, Cyprien Gaillard, etc. His texts have been published in Iskusstvo Kino, Spectate, Colta.ru, Syg.ma and other media. Since 2017, he has been working as a curator at the Moscow International Experimental Film Festival (MIEFF). Currently, he is also working as a film curator at Pushkin House, London.
Everything with AI has grown exponentially this year, and this week we show you AI for animation using different techniques as well as AR, VR and voice cloning. It is astonishing that some of these tools are already a part of our creative toolset, as illustrated in our highlighted projects by GUNSHIP and Fabien Stelzer. Of course, any new toolset comes with its discontents, and so we cover some of those we’ve picked up on this past month too. It is certainly fair to say there are many challenges with this emergent creative practice but it appears these are being thought through alongside the developing applications by those using it… although, of course, legislation is far from here.
Animation
Text-to-image generator Stable Diffusion raised $100M in October this year and is about to release its animation API. On 15 November it released DreamStudio, the first API on its web platform of future AI-based apps, and on 24 November it released Stable Diffusion 2.0. The animation API, DreamStudio Pro, will be a node-based animation suite enabling anyone to create videos, including with music, quickly and easily. It includes storyboarding and is compatible with a whole range of creative toolsets such as Blender, potentially making it a new part of the filmmaking workflow bringing imagination closer to reality without the pain, or so it claims. We’ll see about that shortly no doubt. And btw, 2.0 has higher resolution upscaling options, more filters on adult content, increased depth information that can be more easily transformed into 3D and text-guided in-painting which helps to switch out parts of an image more quickly. You can catch up with the announcements on Robert Scoble’s Youtube channel here –
As if that isn’t amazing enough, Google is creating another method for animating using photographs, think image-to-video, called Google AI FLY. Its approach will make use of pre-existing methods of in-painting, out-painting and super resolution of images to animate a single photo, creating a similar effect to nerf (photogrammetry) but without the requirement for many images. Check out this ‘how its done’ review by Károly Zsolnai-Fehér on the Two Minute Papers channel –
For more information, this article on Petapixel.com‘s site is worth a read too.
And finally this week, Ebsynth by Secret Weapon is an interesting approach that uses a video and a painted keyframe to create a new video resembling the aesthetic style used in the painted frame. It is a type of generative style transfer with an animated output that could only really be achieved in post production but this is soooo much simpler to do and it looks pretty impressive. There is a review of the technique on 80.lv’s website here and an overview by its creators on their Youtube channel here –
We’d love to see anyone’s examples of outputs with these different animation tools, so get in touch if you’d like to share them!
AR & VR
For those of you into AR, AI enthusiast Bjorn Karmann also demonstrated how Stable Diffusion’s in-painting feature can be used to create new experiences – check this out on his Twitter feed here –
Made a quick AR inpainting experiment called We See!
For those of you into 360 and VR, Stephen Coorlas has used MidJourney to create some neat spherical images. Here is his tutorial on the approach –
Also Ran?
Almost late to the AI generator party (mmm….), China has released ERNIE-ViLG 2.0 by Baidu, a Chinese text-to-image AI which Alan Thompson claims is even better than DALL-E and Stable Diffusion albeit using much a smaller model. Check out his review which certainly looks impressive –
Voice
NVidia has done it again – their amazing Riva AI clones a voice using just 30 minutes of voice samples. The application of this is anticipated to be conversational virtual assistants, including multi-lingual assistants and its already been touted as frontrunner with Alexa, Meta and Google – but in terms of virtual production and creative content, it is also possible it could be used to replace actors when, say, they are double booked or poorly. So, make sure you get that covered in your voice-acting contract in future too.
Projects
We found a couple of beautiful projects that push the boundaries this month. Firstly GUNSHIP’s music video is a great example of how this technology can be applied to enhance their creative work. Their video focusses on the aesthetics of cybernetics (and is our headline image for this article). Nice!
Secondly, an audience participation film by Fabien Stelzer which is being released on Twitter. The project uses AI generators for image and voice and also for scriptwriting. After each episode is released, viewers vote on what should happen next which the creator then integrates into the subsequent episode of the story. The series is called Salt and its aesthetic style is intended to be 1970s sci-fi. You can read about his approach on the CNN Business website and be a part of the project here –
Last month we considered the disruption that AI generators are causing in the art world and this month its the film industry’s turn. Just maybe we are seeing an end to Hollywood’s fetish with Marvellizing everything or perhaps AI generators will result in extended stories with the same old visual aesthetic, out-painted and stylized… which is highly likely since AI has to be trained on pre-existing images, text and audio. In this article, Pinar Seyhan Demirdag gives us some thoughts about what might happen but our experience with the emergence of machinima and its transmogriphication into virtual production (and vice versa) teaches us that anything which cuts a few corners will ultimately become part of the process. In this case, AI can be used to supplement everything from concept development, to storyboarding, to animation and visual effects. If that results in new ideas, then all well and good.
When those new ideas get integrated into the workflow using AI generators, however, there is clearly potential for some to be less happy. This is illustrated by Greg Rutkowski, a Polish digital artist whose aesthetic style of ethereal fantasy landscapes is a popular inclusion in text-to-image generators. According to this article in MIT Technology Review, Rutkowski’s name has appeared on more than 10M images and used as a prompt more than 93,000 times in Stable Diffusion alone – and it appears that this is becasue data on which the AI has been trained includes ArtStation, one of the main platforms used by concept artists to share their portfolios. Needless to say, the work is being scaped without attribution – as we have previously discussed.
What’s interesting here is the emerging groundswell of people and companies calling for legislative action. An industry initiative has formed and is evolving rapidly, spearheaded by Adobe in partnership with Twitter and the New York Times called Content Authentication Initiative. CAI aims to authenticate content and is a publishing platform – check out their blog here and note you can become a member for free. To date, it doesn’t appear that the popular AI generators we have reviewed are part of the initiative but it is highly likely they will at some point, so watch this space. In the meantime, Stability AI, creator of Stable Diffusion, is putting effort into listening to its community to address at least some of these issues.
Of course, much game-based machinima will immediately fall foul of such initiatives, especially if content is commercialized in some way – and that’s a whole other dimension to explore as we track the emerging issues… What of the roles of platforms owned by Amazon, Meta and Google, when so much of their content is fan-generated work? And what of those games devs and publishers who have made much hay from the distribution of creative endeavour by their fans? We’ll have to wait and see but so far there’s been no real kick-back from the game publishers that we’ve seen. The anime community in South Korea and Japan has, however, collectively taken action against a former French game developer, 5you. The company used a favored artist’s work, Jung Gi, to create an homage to his practice and aesthetic style after he had died but the community didn’t agree with the use of an AI generator to do that. You can read the article on Rest of World’s website here. Community action is of course very powerful and voting with feet is something that invokes fear in the hearts of all industries.
This week’s pick is a 360 music video – a ‘metaverse’ video – by a creator we’ve been following all year, Jae Solina aka JSFilmz. The film has been created in UE5 and includes some nifty mocap, great dance moves and some interesting lighting effects. Hear what the team have to say about the film and format and let us have your comments too!
YouTube Version of this Episode
Show Notes and Links
Metaverse Music Video, released 10 Sept 2022 (note, the video can be viewed as a VR experience or a 360 video) – where is the Batman Easter Egg?!!!
Our discussion on Friedrich Kirschner’s immersive machinima, person2184, in THIS episode
Nightmare Puppeteer allows 360 filmmaking – check out the engine on Steam HERE
Key questions: what new language might be needed for machinima story vs experience creators to get the most out of VR/360 formats?
Credits –
Speakers: Ricky Grove, Damien Valentine, Tracy Harwood (MIA Phil Rice, courtesy of Hurricane Ian) Producer/Editor: Damien Valentine Music: Scott Buckley – www.scottbuckley.com.au CC 00
In this episode, Tracy, Ricky, Phil and Damien cover the relevance of Nvidia’s special address at CES for machinima creators, Adobe’s Project Shasta, Kerbal Space Programme, the uptake in VR kit over the Christmas period, growth in machinima, NFTs, Philip Rosedale’s return to the Second Life fold, the nail in RoosterTeeth’s RVB saga, Minecraft’s and Rockstar’s astonishing achievements, Ben Grussi’s history episodes and discuss two great questions posed by our followers: what’s the difference between machinima and animation and what’s the advice for adapting prose to visual media formats.
23:57 Philip Rosedale and the future of Second Life for creators
40:30 Halo Xbox 360 multiplayer servers close – the end of the story for Rooster Teeth’s RVB series?
42:50 Ben Grussi’s history of machinima episodes of the Completely Machinima podcast
44:34 Matthew Loris/Zeke: what are the differences between machinima and animation discussion; Completely Machinima interview with Mr Anymation, Tom Jantol
Recent Comments