We're proud to showcase some recent real-time work with 3D / 4D Gaussian splatting.
In January of 2024 we acquired a full 3DGS commercial license with Inria to allow us to offer 4DGS services to clients around the world.
We've spent the last 10 months developing our custom…
Some of IR's early digital puppetry R&D tests utilizing our "AeonX: S0L" system. WIP
Combined with Reality Capture, Wrap4D from R3DS and the awesome Mush3D for wrinkle and fat simulation.
More motion scan pipeline details here
#digitalcreatures
#ir
#4D
White Light Reference for Machine Learning.
Meet our inhouse tech Doggo "Rolo".
"More Doggo than Doggo"
Since July we've been redesigning our scanning pipeline to work with the new 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria.…
IR's new digital human pipeline. The "InfiniteV2" digital double.
R&D tests utilizing our "AeonX: S0L" system for TRILAT capture + Inciprocal processing + artistry!
More pipeline details on our blog -
#digitalhumans
#virtualhumans
#ir
#inciprocal
Meet our inhouse tech Doggo "Rolo". Cinematic RGB Lighting
"More Doggo than Doggo"
Since July we've been redesigning our scanning pipeline to work with the amazing 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria.
IR's…
Volumetric Capture is Hard
There is no magic solution. Anyone interested in spatial R&D can get access to some awesome datasets online to test with, which can be especially interesting when processed with 3DGS and played back as 4DGS. However you need a ton of compute power.…
More 3DGS R&D incoming..
Visual Effects Compositing of RGB + W Captured Gaussian Splats
Using our custom built RGB + W video scanning system, we captured a static pose of
@henrypearce4D
under varying "baked" RGB + W illumination patterns using HDRI probes as the input into our…
High Quality 3D Gaussian Splats in Motion
Here's another X thread to show our early test results with some motion scanned data that we captured back in 2020.
Thanks to Inria's 3D Gaussian Splatting we can now re-process and re-evaluate our motion…
Enter the SPLATRIX!
We digitized
@henrypearce4D
and converted him over to 4DGS to be played back in real-time.
3D Gaussians Splats in Motion. 4DGS.
Our goal was to improve the quality of our AeonX & 4DGS pipeline, with automated background removal to retain fine hair details…
"More Doggo than Doggo"
Motion scanned @ 60fps. Playing back @ 20fps.
Meet our inhouse tech Doggo "Rolo".
Since July we've been redesigning our scanning pipeline to work with the new 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria.
GS has…
Visual Effects Compositing of RGB + W Captured Gaussian Splats Part2
3D Gaussian Splatting
IR + AeonX
Showcasing 4x R&D lighting tests of
@henrypearce4D
lit with our physical lighting rig using the HDRI's listed below
Uffizi - …
Testing out a new tool in our custom Sibr viewer to give
@henrypearce4D
a 3D Gaussian Splatting virtual haircut!
Very useful for removing unwanted splats in real-time in situ and saving to disk.
We now also have full background transparency support and image sequence rendering…
Visual Effects Compositing of RGB + W Captured Gaussian Splats Part3
3D Gaussian Splatting
IR + AeonX
Showcasing R&D tests of
@henrypearce4D
lit with our physical lighting rig using our own custom GI+normals process & 2x baked custom lighting states from…
Presenting a sneak peek @ IR's latest RGB motionscan lighting R&D from 2022. Showcasing high-speed, multiple synchronized video streams of a single performance - a great fit for motion scanning, machine learning and deep fake projects..🧵
#colourscience
#ml
#deepfake
#RGBW
3D Gaussians Splats in Motion. 4DGS/RGB+W.
This post represents our next phase of progress with 4DGS research.
We're working with the original code base from Inria:
Captured with IR's custom built "AeonX:S0L" scanning rig.
The next phase of R&D for us…
Whilst we figure out a solution for temporal filtering on 3D Gaussian Splatting input frames. We can use a temporal filtering plugin on the screen captures or rendered images to act as a visualization tool to check what the future results might look like.
"Processing and…
Dynamic Splats Merging
3DGS Spatial Yoga Training?
Here we present 3x Dynamic Gaussian Splat's together in one scene.
This video show you the current performance we can achieve with 3x different motion scanned 3DGS loaded into one scene. Note these have been reduced down to…
Visual Effects Compositing of RGB + W Captured Gaussian Splats PartV4a
3D Gaussian Splatting
IR + AeonX
For these set of tests we wanted to push the pipeline further:
1) Fully automated ML image segmentation (background removal and floaters) ready for…
"Animatable Gaussians: Learning Pose-dependent Gaussian Maps
for High-fidelity Human Avatar Modeling"
It's great to see 4D data that was captured on IR's "AEONX: DƐUS" rig back in 2022 out in the wild and being used to further advance digital human…
With machine vision cameras we can now scan 18 different lighting states in 0.4 seconds. Or 172 lighting states (scans) in 4 seconds. Using the IDA-Tronic LED nodes. No camera vibrations. No subject movement. No buffer issues. Instant to disk.
#ir
#aeonx
#idatronic
#wecangofaster
3DGS Digital Double Lookdev Reference Part1
In 2019, we captured detailed data of a model from BMA Models in London, illuminating them with our bespoke 156x LED rig. This was achieved using specialized techniques and IR's earlier 48x Ximea camera setup.
With the advancements…
3DGS Digital Double Lookdev Reference Part2
In 2019, we captured detailed data of a model from BMA Models in London, illuminating them with our bespoke 156x LED rig. This was achieved using specialized techniques and IR's earlier 48x Ximea camera setup.
With the advancements…
🎉 IR's AeonX: DƐUS motion scanning system was chosen to capture the ActorsHQ data set for Synthesia IO's HumanRF R&D project
#SIGGRAPH2023
🎉
Project Page:
Thanks to Jon Starck for the oppurtunity to work on HumanRF with Synthesia IO's team 🧵
We plan to post a few videos of some of the challenges that we have to overcome when processing with the 3D Gaussian Splatting technique from Inria. It's incredible but will require some additional work on the processing and playback side to achieve what we want to achieve.
One…
Here's another expression viewed in real-time in the Sibr viewer! This 3DGS expression set used our new ML image segmentation approach to remove unwanted background data and floaters!
Huge congrats to the guys over at R3DS. They've just released Wrap4D! It's quite incredible. Wrap4D contains some 4D sample data captured exclusively with IR's AEONX 4D Motion Scanning System
#wrap4d
#ir
#aeonx
#idatronic
IR's new digital human pipeline. The "InfiniteV2" digital double.
R&D tests utilizing our "AeonX: S0L" system for TRILAT capture + Inciprocal processing + artistry!
More pipeline details on our blog -
#digitalhumans
#virtualhumans
#ir
#inciprocal
Din Djarin & Grogu (Collectibles).
Cinematic RGB Lighting Just look at that shiny beskar!
This is R&D research for a new type of online product viewer! Testing with the awesome Din Djarin & Grogu from
@collectsideshow
#spatial
Imagine using the AeonX system & gaussian…
IR's new digital human pipeline. The "InfiniteV2" digital double.
R&D tests utilizing our "AeonX: S0L" system for TRILAT capture + Inciprocal processing + artistry!
More pipeline details on our blog -
#digitalhumans
#virtualhumans
#ir
#inciprocal
Here's our next "3DGS" X post!
Before we share some of the cool stuff that we've been processing regarding "high quality 3DGS in motion". Some of you might find it interesting to see gaussian splat iterations forming over time?
Watch till the end to see the splats appearing…
We've been working with David Miranda for the last 3 months on developing a Skin shader for Unity, using Jorge Jimenez' research as inspiration! No HDRP using old methods. Comparing triplegangers data with Epic's data set, inside Unity. Blog post to come
#ue4
#battleroyale
#unity
Some of IR's early digital puppetry R&D tests utilizing our "AeonX: S0L" system. WIP. Needs more texture work!
Combined with Reality Capture, Wrap4D from R3DS.
This is literally a scanned face mask "worn as skin"!!!
1000% vampyre!!!
Anyone involved in the digital double industry, whether it's 3D or 4D scanning, it's always worth reflecting on and respecting who stood before us. Standing on the shoulders of giants:
George Borshukov & Team. UCAP for the Matrix. Pre 2000's!
Showcasing the AeonX "Perseus" 360° Motion Scanning System. It's taken us years of R&D (since 2014) to get to this point. We can capture between 60fps and 120fps and output up to 16K textures. We're open to service work.
#aeonx
#motionscanning
#60fps
#4d
For anyone curious regarding the processing times for the 2x 300 frame 3D/4DGS clips we posted.
Yes, you can use 2x RTX4090. It will just take you 4 days (GPU's running 24/7). This is the processing power (energy) required for good quality 2K 4DGS on a small clip of just 20…
@HaywireDhanish
@Tsunduko
On a small clip like this with just 20 cameras yes. @ 30,000 iterations roughly =
300 frames (per 4DGS clip)
1 frame = 20 minutes
3 frames = 1 hour
72 frames = 24 hours
300 frames = 4 days (roughly)
600x frames in total (as shown) would take 8 days on a single RTX4090 or 4 days…
138 GB volumetric capture sequence reviewed in real-time at 60 FPS using
@NiraOfficial
. Data captured and provided by AeonX Motion Scanning System by
@8Infinite8
. Each frame is a distinct mesh and texture. yt:
Here's a sneak peek at IR's AeonX 4D scan data processed using R3DS Wrap4D(Beta)! Captured with our 56x machine vision 60fps system (50GB/s!) The system is designed to capture human subjects for machine learning applications and VFX!
@3dartistchoihun
Colour consistancy and flicker is not an issue with traditional videogrammetry with Reality Capture of Metashape (see the video). What we are talking about in the thread is something else called Gaussian Splatting.
If you want the best cameras and hardware in the business!? If you're looking to evolve from photogrammetry to videogrammetry.. reach out to Ximea
@ximeagmbh
!! They are THE best camera company we've ever worked with. Incredible support and incredible hardware.
Since July we've been redesigning our scanning pipeline to work with the new 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria.
We're testing out some toy captures for real-time product visualization. Seeing what's possible.…
Here's an example of a RAW INGP NeRF without temporal stability. This was one of our first NeRF's in motion and it helped us work out a custom pipeline to align, export and process fixed camera data inside INGP!
#NVIDIA
#INGP
#NeRF
#AEONX
#music
#youth83
It's finally cooked! 110 hours. 1kg of PLA filament (nearly ran out!) Not perfect but usable for 3D/4D scan calibration. If you can get the 1st print layer down.. you're winning.
#infinite
#clones
#notquitewestworldyet
Our paper on scene level inverse rendering from single uncontrolled images, InverseRenderNet, is appearing
@cvpr2019
next week. Paper here: Code and pretrained network now online: Go forth and inverse render!
We can capture 100's of different lighting states to render match too. Or we can interleave dozens of lighting states into a single captured performance.
If you have any questions, feel free to contact us.
Thanks for reading!🙏
For this dataset we used our new "lasso" tool to cut and trim each baked 3DGS by hand.
We've since had a new ML image segmentation process developed that we can scale up across 1000's of expressions to remove background floaters.
Our system is ready for action. Please get in touch if you need our human scanning services, or if you're interested in human NeRF research.
You can find out more information on our new website:
#nerf
#digitalhumans
#motionscanning
#machinelearning
Here's another 3DGS in motion. This one was not as performant due to more of the stage "splats" being visible. Thing of the type of body in motion analysis that will be possible in the future for things like sports and dance. (🧵2/4)
@GozukaraFurkan
The images are aligned in either Reality Capture, or Metashape, or Colmap. They are trained with the Guassian Splatting process then viewed in a modified version of Sibr.
"More Doggo than Doggo"
Motion scanned @ 60fps. Playing back @ 20fps.
Meet our inhouse tech Doggo "Rolo".
Since July we've been redesigning our scanning pipeline to work with the new 3D Gaussian Splatting for Real-Time Radiance Field Rendering method from Inria.
GS has…