Runway

S4 E133 Machinima News Omnibus (Jun 2024)

Tracy Harwood Podcast Episodes June 12, 2024 Leave a reply

This week we celebrate Nvidia’s investment in machinima, discuss latest AIs and give you the heads up on some great projects we want to highlight – there are just too many for us to fully review everything we’re seeing that we want to share with you these days! Do check them out, we’d love to hear your thoughts too – links and notes below.



Youtube Version of This Episode

Show Notes & Links

Homage to Nvidia’s Omniverse Machinima app by Pekka Varis, released 14 April 2024 –

Here’s a link to Omniverse USD Composer

Projects Worth Catching Up On

Endgame by Peaches Chrenko and Dark Machine Audio, sound track –

JP Ferre’s DCS: Spitfires – Cinematic, a real tribute to those brave souls in WWII –

TheDavedood (Scratby Films) animation for Pink Floyd’s Dark Side of the Moon 50 years celebration – this one is for the single, Time –

Anomidae’s latest episode of the Half Life supernatural series Interloper –

Fallout inspired videos: JT Music’s Fallout rap, All in With the Fallout –

and Fallout – Dream on (tribute) by Couch Patrol –

Fables of the Foolish by Dreeko –

AI is Genie-us Init?

ElevenLabs has released a dubbing toolset tutorial, making different languages for videos even more accessible – link here

Google DeepMind has announced a new video generation model called Veo, which produces 1080p res videos for over a minute length in a whole range of different cinematic styles – link here

Stability AI has launched Stable Artisan to a wider user group on Discord – ats a tool for media generation and editing – link here

Showrunner by The Simulation, text to episode generator – link here

Winner of the 2nd AI Film Festival at Runway is by Daniel Antebi, called Get Me Out –

Luc Shurgers’ Skibidi Sam, video only on LinkedIn, created using Replikant – link here

Sony backs out of AI training with its catalogue – article here

Cor Blimey!

The real Backrooms discovered – check it out here

Another movies game, yes really! This one is called Movies Tycoon, link here

We are living in the Matrix, Black Mirror style – check out the episode Joan is Awful –

S4 E102 mAIchinima: Nina & Prelude to Dust (Nov 2023)

Tracy Harwood Podcast Episodes November 9, 2023 Leave a reply

Not strictly machinima, but something we’ve named mAIchinima! This week, we discuss two films using generative AI to create narrative works. In both cases, the techniques employed emphasize the sound – music or voice acting – but whilst one is intentional, the other is not. We share our thoughts on these works and discuss some of the current limitations and benefits observed, which leads us into a timely discussion about style in filmmaking. We also discuss some recent developments in AI for creatives, such as the role of Glaze masking and Nightshade corrupting tools.



YouTube Version of this Episode

Show Notes & Links

Our films this week –

Nina | Denvery Pluto | Episode 2 by Dean Corrigan, released 13 Sept 2023

Prelude to Dust by Dark Machine Audio, released 5 Sept 2023

The generative AI tools we mention in our preliminary discussion are Glaze and Nightshade, information about both can be found here. You can also find out more about Glaze and how it works here –

Ricky’s games mentioned: The Lies of P by From Software and Lords of the Fallen, available on Steam.

Recommended USB microphone reviewed by Ricky is by J Lab, called Talk, link here.

Artist mentioned is Lynd Ward and film style we discuss is Waltz With Bashir, trailer here –

and another artist is H R Giger, whose work on Alien encompasses a distinct style.

Tech Update 1: AI Generators (Apr 2023)

Tracy Harwood Blog April 3, 2023 Leave a reply

March was another astonishing month in the world of AI genies with the release of exponentially powerful updates (GPT4 released 14 March; Baidu released Ernie Bot on 16 March), new services and APIs. It is not surprising that by the end of the month, Musk-oil is being poured over the ‘troubling waters’ – will it work now the genie is out of the bottle? Its anyone’s guess and certainly it seems a bit of trickery is the only way to get it back into the bottle at this stage.

Rights

More importantly, and with immediate effect, the US Copyright Office issued a statement on 16 March in relation to the IP issues that have been hot on many lips for several months now: registrations pertaining to copyright are about the processes of human creativity, where the role of generative AI is simply seen as a toolset under current legal copyright registration guidance. Thus, for example, in the case of Zarya of the Dawn (refer our comments in the Feb 2023 Tech Update), whilst the graphic novel contains original concepts that are attributable to the author, the use of images generated by AI (in the case of Zarya, MidJourney) are not copyrightable. The statement also makes it clear that each copyright registration case will be viewed on its own merit which is surely going to make for a growing backlog of cases in the coming months. It requires detailed clarification of how generative AI is used by human creators in each copyright case to help with the evaluation processes.

The statement also highlights that an inquiry into copyright and generative AIs will be undertaken across agencies later in 2023, where it will seek general public and legal input to evaluate how the law should apply to the use of copyrighted works in “AI training and the resulting treatment of outputs”. Read the full statement here. So, for now at least, the main legal framework in the US remains one of human copyright, where it will be important to keep detailed notes about how creators generated (engineered) content from AIs, as well as adapted and used the outputs, irrespective of the tools used. This will no doubt be a very interesting debate to follow, quite possibly leading to new ways of classifying content generated by AIs… and through which some suggest AIs as autonomous entities with rights could become recognized. It is clear in the statement, for example, that the US Copyright Office recognizes that machines can create (and hallucinate).

The complex issues of the dataset creation and AI training processes will underpin much of the legal stances taken and a paper released at the beginning of Feb 2023 could become one of the defining pieces of research that undermines it all. The research extracted near exact copyrighted images of identified people from a diffusion model, suggesting that it can lead to privacy violations. See a review here and for the full paper go here.

In the meantime, more creative platforms used to showcase creative work are introducing tagging systems to help identify AI generated content – #NoAI, #CreatedWithAI. Sketchfab joined the list at the end of Feb with its update here, with updates relating to its own re-use of such content through its licensing system coming into effect on 23 March.

NVisionary

Nvidia’s progressive march with AI genies needs an AI to keep up with it! Here’s my attempt to review the last month of releases relevant to the world of machinima and virtual production.

In February, we highlighted ControlNet as a means to focus on specific aspects of image generation and this month, on 8 March, Nvidia released the opposite which takes the outline of an image and infills it, called Prismer. You can find the description and code on its NVlabs GitHub page here.

Alongside the portfolio of generative AI tools Nvidia has launched in recent months, with the advent of OpenAI’s GPT4 in March, Nvidia is expanding its tools for creating 3D content –

It is also providing an advanced means to search its already massive database of unclassified 3D objects, integrating with its previously launched Omniverse DeepSearch AI librarian –

It released its cloud-based Picasso generative AI service at GTC23 on 23 March, which is a means to create copyright cleared images, videos and 3D applications. A cloud service is of course a really great idea because who can afford to keep up with the graphics cards prices? The focus for this is enterprise level, however, which no doubt means its not targeting indies at this stage but then again, does it need to when indies are already using DALL-E, Stable Diffusion, MidJourney, etc. Here’s a link to the launch video and here is a link to the wait list –

Pro-seed-ural

A procedural content generator for creating alleyways has been released by Difffuse Studios in the Blender Marketplace, link here and see the video demo here –

We spotted a useful social thread that highlights how to create consistent characters in Midjourney, by Nick St Pierre, using seeds –

and you can see the result of the approach in his example of an aging girl here –

Animation

JSFilmz created an interesting character animation using MidJourney5 (which released on 17 March) with advanced character detail features. This really shows its potential alongside animation toolsets such as Character Creator and Metahumans –

Runway’s Gen-2 text-to-video platform launched on 20 March, with higher fidelity and consistency in the outputs than its previous version (which was actually video-to-video output). Here’s a link to the sign-up and website, which includes an outline of the workflow. Here’s the demo –

Gen-2 is also our feature image for this blog post, illustrating the stylization process stage which looks great.

Wonder Dynamics launched on 9 March as a new tool for automating CG animations from characters that you can upload to its cloud service, giving creators the ability to tell stories without all the technical paraphenalia (mmm?). The toolset is being heralded as a means to democratize VFX and it is impressive to see that Aaron Sims Creative are providing some free assets to use with this and even more so to see none other than Steven Spielberg on the Advisory Board. Here’s the demo reel, although so far we’ve not found anyone that’s given it a full trial (its in closed beta at the moment) and shared their overview –

Finally for this month, we close this post with Disney’s Aaron Blaise and his video response to Corridor Crew’s use of generative AI to create a ‘new’ anime workflow, which we commented on last month here. We love his open-minded response to their approach. Check out the video here –

Tech Update 1 (Nov 2022)

Tracy Harwood Blog October 30, 2022 Leave a reply

Hot on the heels of our discussion on AI generators last week, we are interested to see tools already emerging that turn text prompts into 3D objects and also film content, alongside a tool for making music too. We have no less than five interesting updates to share here – plus a potentially very useful tool for rigging the character assets you create!

Another area of rapidly developing technological advancements is mo-cap, especially in the domain of markerless which lets face it is really the only way to think about creating naturalistic movement-based content. We share two interesting updates this week.

AI Generators

Nvidia has launched an AI tool that will generate 3D objects (see video). Called GET3D (which is derived from ‘Generate Explicit Textured 3D meshes’), the tool can generate characters and other 3D objects, as explained by Isha Salian on their blog (23 Sept). The code for the tool is currently available on Github, with instructions on how to use it here.

Google Research with researchers at the University of California, Berkeley are also working on similar tools (reported in Gigazine on 30 Sept). DreamFusion uses NeRF tech to create 3D models which can be exported into 3D renderers and modeling software. You can find the tool on Github here.

DreamFusion

Meta has developed a text-to-video generator, called Make-A-Video. The tool uses a single image or can fill in between two images to create some motion. The tool currently generates five second videos which are perfect for background shots in your film. Check out the details on their website here (and sign up to their updates too). Let us know how you get on with this one too!

Make-A-Video

Runway has released a Stable Diffusion-based tool that allows creators to switch out bits of images they do not like and replace them with things they do like (reported in 80.lv on 19 Oct), called Erase and Replace. There are some introductory videos available on Runway’s YouTube channel (see below for the Introduction to the tool).

And finally, also available on Github, is Mubert, a text-to-music generator. This tool uses a Deforum Stable Diffusion colab. Described as proprietary tech, its creator provides a custom license but says anything created with it cannot be released on DSPs as your own. It can be used for free with attribution to sync with images and videos, mentioning @mubertapp and hashtag #mubert, with an option to contact them directly if a commercial license is needed.

Character Rigging

Reallusion‘s Character Creator 4.1 has launched with built in AccurRIG tech – this turns any static model into an animation ready character and also comes with cross-platform support. No doubt very useful for those assets you might want to import from any AI generators you use!

Motion Capture Developments

That every-ready multi-tool, the digital equivalent of the Swiss army knife, has come to the rescue once again: the iPhone can now be used for full body mocap in Unreal Engine 5.1, as illustrated by Jae Solina, aka JSFilmz, in his video (below). Jae has used move.ai, which is rapidly becoming the gold standard in markerless mocap tech and for which you can find a growing number of demo vids showing how detailed movement can be captured on YouTube. You can find move.ai tutorials on Vimeo here and for more details about which versions of which smart phones you can use, go to their website here – its very impressive.

Another form of mocap is the detail of the image itself. Reality Capture has launched a tool that you can use to capture yourself (or anyone else or that matter, including your best doggo buddy) and use the resulting mesh to import into Unreal’s MetaHuman. Even more impressive is that Reality Capture is free, download details from here.

We’d love to hear how you get on with any of the tools we’ve covered this week – hit the ‘talk’ button on the menu bar up top and let us know.