This week’s update is all about the virtual production pipeline and digital cultural history.
DaVinci Resolve 18.5 (and .1 fixes) has finally released, and Blackmagic Design have a comprehensive support centre you can make use of here (only for the pro version license holders). The version includes a bunch of new features for integrating AI genie content and collaboration. Here’s an overview, courtesty of MrAlexTech –
Unreal Engine has an ever-expanding and truly talented community. In this tut, Jonathan Winbush (our feature image this week) shares his approach to creating procedurally generated towns using PCG and blueprints inside UE and Cargo (Kitbash3D). Winbush has a wealth of material on his channel, all free, for anyone to pick up and work with, so there’s really no excuse not to learn Unreal Engine –
Boundless Entertainment has release a course for filmmaking, pre-viz and VFX. Its designed for taking beginners to more professional levels in 10 days… mmm, lets see! Its not free, like many of the YouTube tutorials, but for $180 it will undoubtedly suit some learning styles.
Finally, if you want to share your VP process and also learn from others, Nvidia has a new #StartToFinish challenge running til the end of August. Its focussed on those working with the Omniverse platform, with a chance to be showcased on their social media channels. You can find out more about it on their Discord server.
Digital Culture History
We were interested to see a post on the BBC’s website that reported on NoClip’s Danny O’Dwyer rescue of hundreds of hours worth video content of gaming history from landfill. The collection mostly pre-dates YouTube, and comprises of footage and media clips that were cut from being shown on TV or websites. You can see Danny talk about his gold strike here –
We look forward to seeing what Danny digs up as he goes through the material over the next 10 years or so.
Back to the Future, that classic 1980s trilogy we all love for a whole range of reasons, is BACK again. This time, its as a Musical at the Alephi Theatre in London’s West End and the Winter Garden Theatre on Broadway, and in 2024, a North American tour. Its also fascinating to hear the rejection story of Bob Gale and Robert Zemeckis’ original film script – rejected over 40 times before finally being signed. There are certainly many lessons in here for creatives today, not least the process of adapting film FX to theatre, for which MoveAI/Disguise for mocap and virtual production techniques are being employed –
A first in the UK, with a 5G screen test for a dual-location virtual production method for real-time performance capture –
We’ve been following the debate on copyright, fair use and transformative use of IP for what seems like 30 years in the world of machinima (see some of our posts here, here and here) – oh, actually its 27 years…! On 18 May, the world was exercised a little further on the issue of transformative use when the Supreme Court (US) reached its decision on Andy Warhol’s use of a photograph of Prince in a magazine – a case that’s been running since 2016, following Prince’s death. Many suggested this decision is the beginning of end of transformative use – or at least ‘narrows the ‘fair use’ doctrine‘ – and will have massive detrimental impacts on all things created, such as machinima from games engines… however, with the particular scenario fully outlined, this was probably the right outcome for this case. The scenario relates to an unattributed use of an image from a private collecton of works (created and held by Warhol/foundation), where other works involving the same creatives in the collection had previously been attributed and the photographer recompensed when having been used in magazines, and the fact that both Warhol and the photographer (Lynn Goldsmith) made money from selling images individually. So, this decision is about context of use involving the individuals as much as it is ‘fair use’ per se. Justice Sotomayor stated the important factor in the fair-use analysis was that “the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes” pushed the decision in favour of the photographer, arguing that “licenses, for photographs or derivatives of them, are how photographers like Goldsmith make a living. They provide an economic incentive to create original works, which is the goal of copyright.” You can read the ruling in full here – or use your favorite search tool for a link to any one of the numerous news articles covering the case.
So, until such time as the principle applied in this case is actually applied to a creator context, where income is rarely a goal of productions beyond individual recognition and perhaps the meagre YouTube % share for eyeballs it receives, and transformation is generally well beyond that originally intended by say a game dev, it feels like there’s nothing to see here.
On 23 June, Second Life turns 20 years old! There will be virtual parties, exhibitions, product sales and more – for 20 days of course, and you can find out more on the community website here. Happy Birthday to all the Lindens – the first open world environment to truly embrace metaversal themes.
If you want to catch up on some light reading, then its also worth noting that Wagner James Au’s new book releases a week later on 27 June, called Making a Metaverse that Matters. Au also regularly writes some great updates for what has to be one of the longest-running metaverse blogs. Its called New World Notes, which he founded in 2006. Au was the first metaverse journalist and marketer for SL back in 2003. Links to the book here –
Excited to tease the cover for my book, "Making a Metaverse That Matters"!
In stores next month. Please consider pre-ordering.
Nvidia are releasing a monthly update on its blog of all things Omniverse, including latest advancements for the OpenUSD framework that has so quickly become the gold standard for integrating a wide range of creator tools in a 3D workflow. Here‘s the link to the first part of the ‘Into the Omniverse’ series (our feature image for this post) which includes an overview of an update to the connector for Adobe Substance 3D Painter. Substance 3D releases its latest version 203.0 in mid June. This series is a must follow for all content creators, whether or not you own an RTX!
For those seeking advice on devising a virtual production pipeline, Unreal Engine has helpfully released a visualisation guide here and a nice vid here –
Unreal Engine released version 5.2 on 11 May, which includes some fab new features including a preview of its still in dev Procedural Content Generation framework, enabling creators to populate large scenes more efficiently; Substrate, that supports a greater range of surface appearances such as the opalescent finish showcased in this vid –
an enhanced virtual production set of tools for realtime filmmaking support; enhanced VCam system for multi-camera control; and nDisplay extended support, which is setting the scene for the next version 5.3. A link to the release notes is here.
We also spotted a useful tool in the UE Marketplace albeit pricey at $249 for indies: MetaShoot. It includes lighting and render presets for assistance with creating sophisticated lighting setups in your VP studio, released by VINZI – Code Plugins, link here.
Also super helpful is Kitbash3D’s new Cargo asset browser, including some 10,000 searchable assets. The basic account, which is free, allows you to 1-click upload content to your project and manage the assets you have but for a fee of $65/month, the pro version will let you search and access the full model and media library. Its another layer of cost so do check out the small print.
In comparison to the previous six months, the past month has not exactly been a damp squib but it has certainly revealed a few rather under-whelming releases and updates, notwithstanding Adobe’s Firefly release. We also share some great tutorials and explainers as well as some interesting content we’ve found.
Nvidia and Getty have announced a collaboration that will see visuals created with fully licensed content, using Nvidia’s Picasso model. The content generation process will also enable original IP owners to receive royalties. Here’s the link to the post on Nvidia’s blog.
Microsoft has released its Edge AI image generator, based on OpenAI’s DALL-E generator, into its Bing chatbot. Ricky has tried the tool and comments that whilst the images are good, they’re nowhere near the quality of Midjourney at the moment. Here’s an explainer on Microsoft’s YouTube channel –
Stability AI (Stable Diffusion) has released its SDK for animation creatives (11 May). This is an advancement on the text-to-image generator, although of course we’ve previously talked about similar tools, plus ones that advance this to include 3D processes. Here’s an explainer from the Stable Foundation –
RunwayML has released its Gen 1 version for the iPhone. Here’s the link to download the app. The app lets you use a video from your roll to apply either a text prompt or a reference image or a preset to create something entirely new. Of course, the benefit is that from within the phone’s existing apps, you can then share on social channels at your will. Its worth noting that at the time of writing we and many others are still waiting for access to Gen 2 for desktop!
Most notable of the month is Adobe’s release of Firefly for AdobeVideo. The tool enables generative AI to be used to select and create enhancements to images, music and sound effects, creating animated fonts, graphics and fonts and b-roll content – and all that, Adobe claims, without copyright infringements. Ricky has, however, come across some critics who say that Adobe’s claim that their database is clean is not correct. Works created in Midjourney have been uploaded to Adobe Stock and are still part of its underpinning database, meaning that there is a certain percent (small) of works in the Adobe Firefly database that ARE taken from online artist’s works. Here’s the toolset explainer –
Luma AI has released a plug-in for NeRFs in Unreal Engine, a technique for capturing realistic content. Here’s a link to the documentation and how-tos. In this video, Corridor Crew wax lyrical about the method –
Tuts and Explainers
Jae Solina aka JSFilmz has created a first impressions video about Kaiber AI. This is quite cheap at $5/month for 300 credits (it seems that content equates to appx 35 credits per short vid). In this explainer, you can see Jae’s aged self as well as a cyberpunk version, and the super-quick process this new toolset has to offer –
If you’re sick to the back teeth of video explainers (I’m not really), then Kris Kashtanova has taken the time to generate a whole series of graphic novel style explainers (you may recall the debate around her Zarya of the Dawn Midjourney copyright registration case a couple of months back) – these are excellent and somehow very digestible! Here’s the link. Of course, Kris also has a video channel for her tutorials too, latest one here looks at Adobe’s Firefly generative fill function –
In this explainer, Solomon Jagwe discussed his beta test of Wonder Studio’s AI mocap for body and finger capture although its not realtime unfortunately. This is however impressive and another tool that we can’t wait to try out once its developoer gets a link out to all those that have signed up –
There has been a heap of hype about an advert created by Coca Cola using AI generators (we don’t know which exactly) but its certainly a lot of fun –
In this short by Curious Refuge, Midjourney has been used to re-imagine Lord of the Rings… in the style of Wes Anderson, with much humor and Benicio del Toro as Gimli (forever typecast and our feature image for this post). Enjoy –
We also found a trailer for an upcoming show, Not A Normal Podcast, but a digital broadcast where it seems AIs will interview humans in some alternative universe. Its not quite clear what this will be, but it looks intriguing –
although it probably has a way to go to compete with the subtle humor of FrAIsier 3000, which we’ve covered previously. Here is episode 4, released 21 March –
This week, tech updates cover Epic’s new tools for self-publishing, Omniverse’s USD rebrands, thoughts about the nascent metaverse and some throwbacks to good-old-fashioned machinima creative techniques.
Epic’s Games Store
Surely a move that will make rival Steam squirm, Epic announced on 9 March that it has launched new tools for self-publishing on the Games Store, all on the back of its 68M active monthly users. Publishers will receive 88% of the revenue through sales (compared to 70% on Steam). There are some interesting points raised in the T&Cs, such as the need for cross-playability (across all PC stores), achievement tracking for games, age rating requirements and an affiliate creator programme that enables publishers to share their takings with others – check out the T&Cs on their announcement here. The announcement intimates at much bigger things to come, relating to metaverse propositions, but its an interesting development for now. Here’s a walk through of the tools from their livestream about it –
Nvidia’s Omniverse Create and Omniverse View are rebranding, announced on 3 March. These will now be called, respectively, Omniverse USD Composer and Omniverse USD Presenter. The omnipresence of USD (Universal Scene Description) has become a driving force for 3D creative development in a very short space of time – just last August, Nvidia summarized its vision with embedding USD as the foundation of the metaverse for creatives (and also industrial teams, smart services providers and such), where content could be pushed across a vast array of different platforms. Less than a year later, workflows everywhere have evolved with it and USD is now a ubiquitous technology, much like the internet is the driving force for the web. What’s a little intriguing is why draw attention to it at this juncture, and what’s the point of editing archival videos to include the new names, like this one – recognition, reinforcement, repositioning or something new coming down the pipeline?
Beyond the hype, and clearly the practices as we’ve highlighted above, the metaverse is taking shape in interesting ways. An interesting article, published in VentureBeat on 4 March, highlights the lengths that media and entertainment companies such as Sony are going to in creating virtual worlds that transcend film, game and experiences, including in VR and theme parks. These are more than alignments of creative talent teams, but allude to the potential of vast new ecosystems for collaborators and partners. What’s interesting of course is that the inflection into such ecosystems can be from any creative medium (game, film or artwork presumably), with outputs that are going to be more visceral and consequently more immersive. Since toolsets such as USD facilitate the creation of these ecosystems, it will be interesting to see how indies get in on this action too – we’re already seeing a number of start-up enterprises pushing the boundaries, but there’s also scope for small studios to join in. Question is, where are they now?
Cinematics (the Old Way)
No Man’s Sky has been a machinina creators’ go-to for some time, and this short gives a great overview of how to create cinematics in the environment, by EvilDr.Porkchop (also our blog post feature image) –
Eve Online is another such environment, and now of course a [very] old one, but here’s a nice ‘how to’ for making epic looking machinimas, by WINGSPAN TT –
March was another astonishing month in the world of AI genies with the release of exponentially powerful updates (GPT4 released 14 March; Baidu released Ernie Bot on 16 March), new services and APIs. It is not surprising that by the end of the month, Musk-oil is being poured over the ‘troubling waters’ – will it work now the genie is out of the bottle? Its anyone’s guess and certainly it seems a bit of trickery is the only way to get it back into the bottle at this stage.
More importantly, and with immediate effect, the US Copyright Office issued a statement on 16 March in relation to the IP issues that have been hot on many lips for several months now: registrations pertaining to copyright are about the processes of human creativity, where the role of generative AI is simply seen as a toolset under current legal copyright registration guidance. Thus, for example, in the case of Zarya of the Dawn (refer our comments in the Feb 2023 Tech Update), whilst the graphic novel contains original concepts that are attributable to the author, the use of images generated by AI (in the case of Zarya, MidJourney) are not copyrightable. The statement also makes it clear that each copyright registration case will be viewed on its own merit which is surely going to make for a growing backlog of cases in the coming months. It requires detailed clarification of how generative AI is used by human creators in each copyright case to help with the evaluation processes.
The statement also highlights that an inquiry into copyright and generative AIs will be undertaken across agencies later in 2023, where it will seek general public and legal input to evaluate how the law should apply to the use of copyrighted works in “AI training and the resulting treatment of outputs”. Read the full statement here. So, for now at least, the main legal framework in the US remains one of human copyright, where it will be important to keep detailed notes about how creators generated (engineered) content from AIs, as well as adapted and used the outputs, irrespective of the tools used. This will no doubt be a very interesting debate to follow, quite possibly leading to new ways of classifying content generated by AIs… and through which some suggest AIs as autonomous entities with rights could become recognized. It is clear in the statement, for example, that the US Copyright Office recognizes that machines can create (and hallucinate).
The complex issues of the dataset creation and AI training processes will underpin much of the legal stances taken and a paper released at the beginning of Feb 2023 could become one of the defining pieces of research that undermines it all. The research extracted near exact copyrighted images of identified people from a diffusion model, suggesting that it can lead to privacy violations. See a review here and for the full paper go here.
In the meantime, more creative platforms used to showcase creative work are introducing tagging systems to help identify AI generated content – #NoAI, #CreatedWithAI. Sketchfab joined the list at the end of Feb with its update here, with updates relating to its own re-use of such content through its licensing system coming into effect on 23 March.
Nvidia’s progressive march with AI genies needs an AI to keep up with it! Here’s my attempt to review the last month of releases relevant to the world of machinima and virtual production.
In February, we highlighted ControlNet as a means to focus on specific aspects of image generation and this month, on 8 March, Nvidia released the opposite which takes the outline of an image and infills it, called Prismer. You can find the description and code on its NVlabs GitHub page here.
Alongside the portfolio of generative AI tools Nvidia has launched in recent months, with the advent of OpenAI’s GPT4 in March, Nvidia is expanding its tools for creating 3D content –
It is also providing an advanced means to search its already massive database of unclassified 3D objects, integrating with its previously launched Omniverse DeepSearch AI librarian –
It released its cloud-based Picasso generative AI service at GTC23 on 23 March, which is a means to create copyright cleared images, videos and 3D applications. A cloud service is of course a really great idea because who can afford to keep up with the graphics cards prices? The focus for this is enterprise level, however, which no doubt means its not targeting indies at this stage but then again, does it need to when indies are already using DALL-E, Stable Diffusion, MidJourney, etc. Here’s a link to the launch video and here is a link to the wait list –
A procedural content generator for creating alleyways has been released by Difffuse Studios in the Blender Marketplace, link here and see the video demo here –
We spotted a useful social thread that highlights how to create consistent characters in Midjourney, by Nick St Pierre, using seeds –
and you can see the result of the approach in his example of an aging girl here –
JSFilmz created an interesting character animation using MidJourney5 (which released on 17 March) with advanced character detail features. This really shows its potential alongside animation toolsets such as Character Creator and Metahumans –
Runway’s Gen-2 text-to-video platform launched on 20 March, with higher fidelity and consistency in the outputs than its previous version (which was actually video-to-video output). Here’s a link to the sign-up and website, which includes an outline of the workflow. Here’s the demo –
Gen-2 is also our feature image for this blog post, illustrating the stylization process stage which looks great.
Wonder Dynamics launched on 9 March as a new tool for automating CG animations from characters that you can upload to its cloud service, giving creators the ability to tell stories without all the technical paraphenalia (mmm?). The toolset is being heralded as a means to democratize VFX and it is impressive to see that Aaron Sims Creative are providing some free assets to use with this and even more so to see none other than Steven Spielberg on the Advisory Board. Here’s the demo reel, although so far we’ve not found anyone that’s given it a full trial (its in closed beta at the moment) and shared their overview –
Finally for this month, we close this post with Disney’s Aaron Blaise and his video response to Corridor Crew’s use of generative AI to create a ‘new’ anime workflow, which we commented on last month here. We love his open-minded response to their approach. Check out the video here –