550am December Mix Moment 2023
DJ Set
Motion Capture recorded with Rokoko
Animated and rendered in Unreal Engine
Planning: August/September 2023
Performance Capture recorded November 13-23, 2023
Final sequence uploaded December 21, 2023
This is the fifth DJ set performance animation I’ve created in Unreal Engine. For this animation I wanted to expand upon recording more dance and lipsync setups akin to a proper music video. This would be supplemented by the live performance recording I’ve done for my previous animations. During the live performance recording I am mixing and recording the DJ set that will become the soundtrack for this animation. The intention was to create an homage to 90s MTV style programing and something in between a music video block and a dance show.
Initial concept work for this project began around Summer 2023, with proper work on the concept for the animation and DJ mix beginning in August. The rough version of the DJ mix with the final tracklist was completed early September. Asset, set design, and wardrobe work for this project began September 13. Planning work for the shot list and final wardrobe styling was completed in early October. This is my first project involving a comprehensive list of every camera and performance recording take I wanted, as well as written direction for how I wanted to move and shoot the motion. 12FPS fast motion and 48FPS slow motion shots were also planned and recorded.
My target date was originally November 10 but due to the volume of work required on assets and motion capture I would be running behind schedule by a month and a half. I also delayed the work slightly in hopes I would receive my Rokoko Headrig I ordered specifically to get better face capture in time. This did not happen but the extra time was used to focus on asset work.
Wardrobe for this animation built upon my work from the past year and a half of designing almost all of Neptune’s garments as low poly meshes that are rigged to her skeleton. This made it very easy to reuse a lot of her wardrobe for outfit changes. However a certain number of new clothes needed to be made. In total 20 new pieces including shoes and jewelry were created for this project for Neptune and the Metahuman characters. Each piece was modelled and retopologized in Marvelous Designer and rigged in Maya.
In order to facilitate the 13+ outfit changes for this animation, Neptune was set up as a modular character in Unreal Engine. This involved splitting her geometry into a separate Skeletal Mesh and setting up an Actor Blueprint in UE with each part of her body and rigged clothing set to follow the base skeleton. This creates an issue however for the pieces of clothing that have physics elements. Using the Set Leader Pose component ignores the Physics Asset of the Skeletal Mesh you have set to follow. To fix this I just added the bones for the physics onto the base skeleton and adjusted the base model’s Physics Asset accordingly. This only affects clothing that uses those bones for physics and does not cause issues for other garments.
Adding custom clothing to the Metahuman characters was done in a similar fashion of rigging the low poly clothing onto the base skeleton and using the Set Leader Pose component to have the clothing’s skeleton follow the base skeleton.
I opted not to use any cloth simulation, either offline or realtime. Cloth simulation in UE is often times not worth the headache and the wardrobe was intentionally styled to avoid garments that would need cloth simulation to look correct in motion. Offline simulation would require a lot of processing for each take of character motion and would reduce flexibility in swapping out wardrobe pieces.
Two new hairstyles were also modelled for Neptune. I continue to prefer hair cards in Unreal Engine. Even though the groom system renders hair beautifully, the physics implementation is severely lacking. It’s much easier to just rig hair cards. But in general I also prefer any opportunity that lets me develop my skills further and I’ve been meaning to refine my hair card workflow. I find using Ornatrix in C4D to design both the strand groom reference and the card layout is the most comfortable for me. Textures were created using a mix of FiberShop to create the alphas and Substance Painter to stack the alphas in unique ways and create an alpha texture that could be used for multiple hairstyles.
I chose to present this animation in 4:3 to clearly be evocative of this time period as well as take advantage of the aspect ratio’s framing potential. 4:3 is the perfect aspect ratio for music videos because of how it allows you to frame the face and body with very constrained room for negative space in your framing.
The idea of constraint extends beyond the choice of aspect ratio to set design ideas and camera decisions. In previous work I opted for wider environments and tighter lenses. Camera work as also all keyframed manually which lent itself a motion controlled camera look. For this project I deliberately made the main hallway set very tight. The intention was to create a space that dimensionally mimicked my actual recording space which is a small sliver of space in my bedroom. Camera decisions were informed by this choice. I wanted to shoot handheld with wider lenses (10-30mm) with the exception of a single keyframe moved camera in the center of the set.
The final export of the project was set at 4K. However the ProRes export from Unreal Engine was rendered at 5K. Because of the heavy use of post processing and lens distortion which lowers image quality I had to render higher than my target resolution and downscale for final export. This also smooths any artefacts from the anti aliasing. 24FPS was chosen not just for its cinematic properties but mostly for practical concerns with storage space. Using raw motion capture at 24FPS is risky as it can produce a jittery look. For previous projects I opted for 60FPS to account for jitter in the motion with the trade off of lower resolution. For this project I wanted to prioritize image quality and thus decided on 5K res @ 24FPS. I also find handheld cameras tend to look better at 24FPS vs 60FPS where they tend to look too synthetic and ungrounded.
Color was graded in Davinci Resolve. All shots were rendered in ACESCG. Most shots were rendered as 12bit ProRes 4444. Additional shots were rendered as 10bit ProRes 422 to save storage space. I don’t have the equipment or eyes yet to grade the footage for HDR but it’s something I would like to come back to another time. This was my first time grading ACES footage. While I could have done my coloring entirely in the post processing in Unreal and exported the rec709 footage from there I wanted to create raw exports that could allow for further work and practice in the future as I develop my eyes and taste for coloring video.
Performance capture for this project was recorded between November 13 and November 22. I used a first generation Rokoko Smartsuit, streaming directly into Unreal Engine. And UE Livelink with my iPhone for face capture. I chose to record directly into a Level Sequence inside UE to make processing and labeling the takes simple and I would be exporting the Animation Sequence for the character skeleton and brining it into a different Unreal project file. (There was an issue with the plugin for streaming the mocap data and I had to use two different versions of Unreal to facilitate capture.) I prefer working with individual Level Sequences vs using the Take Recorder (or even using the Take Recorder to record directly to the sequence..) as I find it’s easier to manage both recording and controlling what parameters are being recorded. Recording into a Level Sequence also made syncing audio very very easy. There is only a few frame delay between audio and both the livestreamed and recorded character motion. Unreal Engine naturally has a lot of delay with audio so I cleaned up audio sync for each clip in Premiere Pro instead.
Performance capture for the Metahuman characters was recorded inside of the legacy version of Rokoko Studio and exported as an FBX and retargeted onto the Metahuman skeleton in Maya. Because there were fewer takes to manage and because of the complexity involved with Metahuman characters in Unreal, handling the animation externally was much faster. As a result of the additional processing vs Neptune only using the raw livestream into UE - the male dancers have smoother animation. While I could have used this for Neptune it would have made simultaneous facial capture and lipsync much more difficult.
For this project, like my other live recording work, I used raw motion capture in the final animation. Using raw motion capture exclusively extends beyond the logistics involved with manual clean up and takes on an artistic significance in the work. On my own and within the timeline I set for myself it would be impractical to manually correct and clean up 76 takes of motion capture for Neptune and 47 takes for the dancers. A major theme in my work is interpretation and obfuscation. The raw motion capture is just an imperfect interpretation of my body movement to data. Which is another layer upon the interpretation of how I feel, processed by my brain and sent to my muscles and joints as electrical signals. I wanted a sense of rawness and grit in my work. It’s very easy to add faux grunge in CG work but that comes with the risk of looking like everything else and feeling inauthentic. I find sincerity in showing the rawness of my craft and deliberately leaving things unpolished. The contrasts between real and synthetic, sterile and gritty, stylized and hyper real are all intentional in my work.
Art direction for the project was largely informed by millennium era adult contemporary aesthetics, eurodance music videos, and urban spaces and corridors. Minimal, chic, mature, and informed by late 90s club styles sonically and visually. I wanted background dancers for this project. In order to maintain a high level of visual realism and ease of operation while working in UE I opted to use Metahuman characters with custom wardrobe. Their high level of realism is deliberately meant to contrast Neptune’s stylized form, conversely however in motion Neptune feels more real than the dancers.
Ideas of stillness, existing outside of my own body, and performing not just for others but as a version of myself for myself were the seeds that grew into this project. The urge to prove myself on a technical and production level were also at the forefront during the course of the project. This is both my most introspective work and technically refined and sophisticated as of writing. I wondered about objectivity and how I express myself with so many layers of obfuscation. Contrasting sentiment and the imprecise feelings and memories that come with introspection and stillness with technical and production precision in a way mimics the many contrasting visual motifs in my work.
550am December Mix Moment 2023
Neptune is a Virtual Idol.
Neptune is a Virtual Idol.
Directed / Produced / Mixed by Joshua Keeney (550am)
Kylie Minogue - Too Far (Junior Vasquez Remix)
Britney Spears - I'm A Slave 4 U (The Light Remix)
Pet Shop Boys - Somewhere (Trouser Enthusiasts Mix)
Kanye West - Love Lockdown (Morales Club Mix)
Björk - Violently Happy (Mercurio Remix)
Geri Halliwell - Bag It Up (Trouser Enthusiasts Mix)
Beyoncé - Upgrade U (Peter Rauhofer Reconstruction)
Mousse T. feat. Hot 'n' Juicy - Horny (Boris Gets Horny Mix)
Junior Vasquez meets Fire Island - Get Your Hands Off My Man (A Dub 4 Junior)
Roxette – The Look ’95 (Rapino Club Mix)
Foxy Brown - I'll Be (Foxy Brown Mix)
Jennifer Lopez - If You Had My Love (Metro Club Remix) (Instrumental)
Beyoncé - Run The World (Girls) (Steamweaver Who Run The House Mix)
Britney Spears - I'm A Slave 4 U (The Light Remix)
Pet Shop Boys - Somewhere (Trouser Enthusiasts Mix)
Kanye West - Love Lockdown (Morales Club Mix)
Björk - Violently Happy (Mercurio Remix)
Geri Halliwell - Bag It Up (Trouser Enthusiasts Mix)
Beyoncé - Upgrade U (Peter Rauhofer Reconstruction)
Mousse T. feat. Hot 'n' Juicy - Horny (Boris Gets Horny Mix)
Junior Vasquez meets Fire Island - Get Your Hands Off My Man (A Dub 4 Junior)
Roxette – The Look ’95 (Rapino Club Mix)
Foxy Brown - I'll Be (Foxy Brown Mix)
Jennifer Lopez - If You Had My Love (Metro Club Remix) (Instrumental)
Beyoncé - Run The World (Girls) (Steamweaver Who Run The House Mix)
Performance Capture recorded and performed by Joshua Keeney
Capture recorded with Rokoko / UE Livelink
Capture recorded with Rokoko / UE Livelink
CG / Character Design / Wardrobe Modelling / Styling by Joshua Keeney
Male Dancers created with Metahuman Creator by Epic Games / 3Lateral
Male Dancers created with Metahuman Creator by Epic Games / 3Lateral
Original Garment Designs and Graphics by
Joshua Keeney
Terrell Davis
Tom Ford
Gucci
Mugler
Maison Margiela
Versace
Manolo Blahnik
Jean Paul Gaultier
Marc Jacobs
Vivienne Westwood
David Koma
Van Cleef & Arpels
Diesel
Oakley
Barragan
Joshua Keeney
Terrell Davis
Tom Ford
Gucci
Mugler
Maison Margiela
Versace
Manolo Blahnik
Jean Paul Gaultier
Marc Jacobs
Vivienne Westwood
David Koma
Van Cleef & Arpels
Diesel
Oakley
Barragan
Additional wardrobe for Metahuman characters by Epic Games / 3Lateral
Oakley Juliet model by 3dm-store (cgtrader)
Oakley Juliet model by 3dm-store (cgtrader)
Unreal Marketplace Plugins and Assets:
Rokoko Studio Live - Rokoko Electronics ApS
Dirt Mask - Nest
Laser Show System - xxJulexx
BLOOMIFY- Convolution Bloom pack Vol.1 - Thunderpoly
Convolution Bloom Kernels Pack - Epsilonite
Quixel Megascans
Rokoko Studio Live - Rokoko Electronics ApS
Dirt Mask - Nest
Laser Show System - xxJulexx
BLOOMIFY- Convolution Bloom pack Vol.1 - Thunderpoly
Convolution Bloom Kernels Pack - Epsilonite
Quixel Megascans
Visuals: (c) Joshua Keeney / 550am 2023
550am.net