This is blog post is a follow to Part 1 here about IR’s new scanning pipeline.

We’ve been working closely with Inciprocal for the last few years, designing and specifying out a capture pipeline utilizing our AeonX: S0L scanning rig to fit within the Inciprocal software pipeline.

Our plan is to try and blog post at least once a month!

Today’s inspiration:

[soundcloud url=”https://api.soundcloud.com/tracks/1083639427″ params=”” width=” 100%” height=”166″ iframe=”true” /]

Back in 2010 IR released the Infinite head scan using much more basic scanning processes and a lot of hand sculpting. The scan was downloaded by 1,000’s of artists, games companies and research institutes.

cgchannel.com/2010/09/infinite-realities-releases-free-photorealistic-head-model
github.com/keijiro/InfiniteScan
zbrushcentral.com/t/infinite-3d-head-scan-and-sculpt/295587
docs.unrealengine.com/4.27/en-US/RenderingAndGraphics/Materials/LightingModels/SubSurfaceProfile/

It was even at one point integrated into Unity and Unreal! We’ve since learned a great deal and our scanning system has had a complete overhaul. We’ve finally had a chance to document this process.

You can see an early example of our new R&D digital double research. It’s still WIP and we are learning as we go. We utilized IR’s captured data combined with the Inciprocal full Disney PBR texture stack. We took these processed results added some extra modifications, like extra sculpting and texture work. We then re-built our LED rig virtually and visualized the results inside Marmoset Toolbag 4’s raytraced virtual environment.

Using the data that’s been captured with IR’s AeonX: S0L scanning system, it’s then processed through the Inciprocal pipeline. Once completed it then has some further artistic enhancements added, such as texture tweaks, eye models, teeth and hair details.

For accurate validation we captured a mirror ball set, grey ball set and colour calibration chart inside our capture rig and replicated them digitally.

Split images. Digital Double on the right. Real-World subject on the left.

The new version of Toolbag 4 now features accurate ray traced hard caustics. Instead of using the cornea height bump method and specular trick we built a more realistic eye model featuring a simulated liquid to mimic the aqueous humour inside the anterior chamber of the eye model. This can then better simulate light transport inside the iris and creates some really interesting scattering of light and refractive caustic effects.

This technique also gets rid of the nasty side effects of the cornea/iris bump shadow that you will see if you use a bright direct sunlight using the height bump technique. It will also help bounce light around inside the iris when your digital double is being lit from a glancing angle. The only drawback is that you need a lot of samples to get clean results.

Once the scan, textures and materials have been validated we can then begin to render the subjects in other lighting conditions!

We used CamtrackAR to provide some quick virtual camera tracking data captured from a 3D print of the subject on an iPhone. Converted in Blender then rendered in Toolbag 4!

Another detailed set of videos steady cam again, this time in slow-motion. The digital-double still has further work to be done but time was running out this month to get something published for another blog post!

Our next step is to develop a new optimized capture pipeline to allow us to capture a full Disney PBR stack of expressions to process with Inciprocal. This will allow us to capture detailed wrinkle information at the pore level for the diffuse, specular and roughness properties of a subject. This is still a WIP.

We can then combine this data with our own motion scanned data.

Our next objective is to bring life to IR’s scans. We’ve already started to run experiments utilizing the power from our motion scanning pipeline. Our motion scanning pipeline was the driving force behind the recent Unity Enemies demo and ZIVA ZRT product launch. You can see an early example below. See if you can figure out what the dialogue is from!!?

We’re really excited about the potential of our new AeonX: S0L system and the Inciprocal pipeline. We’re looking forward to working with clients on new and exciting projects and pushing the boundaries of what’s possible with 3D scanning and facial capture technology.

Look out for Part 3 of our Digital Double Pipeline.

Next time will be talking about scanning in “colour”..

Thanks for reading

We’ve been working closely with Inciprocal for the last few years, designing and specifying out a capture pipeline utilizing our AeonX: S0L scanning rig to fit within the Inciprocal software pipeline. Our scanning and digital double sculpting work was recently featured in Unity’s Enemies demo and the ZIVA acquisition presentation.

Today’s inspiration:

[soundcloud url=”https://api.soundcloud.com/tracks/1287497917″ params=”” width=” 100%” height=”166″ iframe=”true” /]

You can see an early example of our R&D digital double research. We utilized IR’s captured data combined with the Inciprocal full Disney PBR texture stack. We took these processed results added some extra modifications, like extra sculpting and texture work. We then re-built our LED rig virtually and visualized the results inside Marmoset Toolbag 4’s raytraced virtual environment.

Split images. Digital Double on the left. Real-World subject on the right.

The more data that can be fed into the Inciprocal software pipeline the better and with as minimal subject movement as possible. IR’s machine vision cameras are essential for this combined with IR’s IDA-Tronic LED system which is lightning fast and highly programmable. Other LED rigs on the market are sometimes limited by the amount of programmable lighting states that they can store. The IDA-Tronic system has the ability to store over 120 different lighting states at any one time, plus as many OLAT states as you require.

This novel capture and processing pipeline is not using a spherical gradient or binary gradient method.

We decided on 36x pairs of cross-polarized and parallel-polarized TRILAT lighting states to be fed into the Inciprocal pipeline. That’s a total of 76x lighting states, plus 2x global illumination states for a total of 78x captures, as well as an additional 156x OLAT shots at the end of the sequence for our main reference cameras. This is required for additional ground truth “unfiltered” reference.

We use a cluster of triplets (TRILAT) to output as much light as possible. We capture at millisecond speeds.

We’ve designed and built a hybrid custom photometric / motion scanning system from the ground up, utilizing 59x 12MP machine vision cameras from Ximea. We use polarizing filters on 56x of our main cameras and no filters on the 3x main cameras cam57, cam58 & cam59.

TRILAT Examples (cam01, cam19 & cam35)

Speed is essential!

We utilize our own triggering system to sync the LED’s and cameras together, as well as a custom software solution called “AeonX” to interface with the cameras.

The LED system comprises 312x LED’s with 3 different controllable sectors (Sector0, Sector1 and Sector2). The whole system is extremely programmable. We utilize polarizing filters on some of the 3 sectors and on some of the cameras.

In a previous Blog post, we showcased our SDL pipeline. When operating in this new “TRILAT Inciprocal” mode we can trigger the S0L system at 62.5 fps. This allows us to capture 78x different lighting states (or scans!) in 1.248 of a second! Compared to our older DSLR system which would trigger at between 3-5fps which was MUCH slower and not reliable, or also in comparison to mirror less Sony cameras that trigger at a slower pace of about 20fps.

We’re able to trigger all of our 59x Ximea machine vision cameras at blazingly fast speeds to acquire thousands of images in under 1.248 seconds. This data is then streamed directly to disk, instantly, for fast playback and observation.

In this mode, to produce a single scan we would typically acquire 4,602 images, at about 55GB’s per expression. Below you can see a subset of 56x cameras in a row, representing each lighting state.

We can also accompany this TRILAT Inciprocal set with a post OLAT spiral set of another 156x “no-filter” illuminations, triggering just the 3x “no-filter” reference cameras. That’s another 468x images.

Why do we need all this data? 1) Because we can! 2) Having as much real-world reference data as possible REALLY helps with the Digital Double look dev validation stage. Whether it’s to check skin reflectance, refractive caustics in the subjects eyes, the anisotropic properties of hair strands, vellus back scattering reference, subsurface scattering reference. The list goes on. The data is invaluable.

We can then re-build our TRILAT images from the OLAT spiral set from the perspective of cam57, cam58 & cam59 (no-filters) to produce a pure unfiltered reference set. The OLAT set and the re-built TRILAT set can then be used as the true render validation for the digital double aspect of the pipeline once Inciprocal has processed and delivered the TRILAT data back to us. This RAW scan will act as the base template for Inciprocal to process with.

Once the scanning session is complete we will build the base photogrammetry scan and provide the RAW scan and calibration data from REALITY CAPTURE to Inciprocal ready for their part of the differential rendering pipeline. After processing at Inciprocal they will then deliver the albedo, reflection, roughness, world space, tangent space and displacement texture maps. We can then use this data to render match to IR’s reference captures and camera alignment from REALITY CAPTURE.

The beauty of the Inciprocal pipeline is the accuracy and detail of the output maps. A full Disney PBR texture stack is reconstructed that usually takes around 40-50 minutes on a single @NVidia RTX 3080. The data is plug and play and acts as great base template to work from to further enhance (if required).

We’re working on optimizing the amount of data captured to be able to support expressions. This is further R&D work to be carried out.

We’re really excited about the potential of our new AeonX: S0L system and the Inciprocal pipeline. We’re looking forward to working with clients on new and exciting projects and pushing the boundaries of what’s possible with 3D scanning and facial capture technology.

Look out for Part 2 of our Digital Double Pipeline..

Thanks for reading

For the last few years, we’ve been busy re-designing our photometric and motion scanning system as well as our pipeline. We’ve not posted much on social media or on our website simply because our systems have been in flux and we’ve been super busy with client work. We’re always pushing and always trying to fine-tune and tweak our systems and the scanning service we provide.

[soundcloud url=”https://api.soundcloud.com/tracks/886560637″ params=”” width=” 100%” height=”166″ iframe=”true” /]

We’ve designed and built a hybrid custom photometric / motion scanning system from the ground up, utilizing 56x 12MP machine vision cameras from Ximea and 3x 24MP Volucam cameras from IOI Industries used purely for reference capture. The system operates in 2 different modes shown below.

“Motion scanning” (60fps)

Photometric/SDL “Sequenced Directional Lighting” (62.5fps)

We utilize our own triggering system and software solution to control and operate the cameras. Our camera system is complemented by the IDA-Tronic LED system

The LED system comprises 156x LED’s with 3 different controllable sectors (Sector0, Sector1 and Sector2) and the whole system is extremely programmable. We utilize polarizing filters on some of the 3 sectors and on some of the cameras.

When operating in SDL mode we can trigger the system at 62.5 fps and this allows us to capture 20 different lighting states (or scans!) in 0.288 of a second! Compared to our older DSLR system which would trigger at between 3-5fps which was MUCH slower and not reliable, or also in comparison to mirror less Sony cameras that trigger at a slow pace of about 20fps.

When we first started building photogrammetry systems back in 2009, the idea of syncing multiple cameras once, reliably, was a massive technical challenge. Now we’re able to do it multiple dozens of times a second, without fail.

Our new system is lightning fast and allows for very rapid image acquisition straight to disk on an operating system, eliminating the issue of subject movement and having to “stitch images” together or use optical flow. Also, because we capture at such high frame rates, we can pulse our LEDs very rapidly in sync with the cameras, which removes the issue of motion blur and also prevents blinding the talent with bright flashlights, like our older single shot “DSLR photogrammetry systems” tended to do. Also, as a bonus.. no exploding flash bulbs!

It’s quite difficult to express and visualize how much data we are able to acquire. So the below images are just a small subset of what’s captured.

Another added bonus of the system, for certain expressions, we can also add an additional set of lighting captures called a “Spiral” set to capture all the required OLAT shots of the subjects from each LED. This will add an additional 156 lighting states, which takes around 2 seconds for a total of 176 images (or scans!) per camera!

All of the 20 image sets below were captured in about 0.288 of a second. To the human eye, it looks like a single flash.

To give you an idea of the amount of image data captured from a single photometric / SDL / OLAT scan = 176 images per camera x 59 cameras = 128GB of data x the number of expressions a project might require and that’s just for the RAW image data. That’s 128GB of raw image data for just one expression.

You can see below a set of captures from our 3 main reference cameras used for render matching later in the pipeline. We make sure to calibrate each cameras exposure and white balance correctly to an xrite chart.

This pipeline is then carried through to the FACS part of the capture process, so the above datasets are acquired for each subjects expression scan. We can scan as many expressions as a client requires.

We can capture a wealth of data rapidly. These data sets are ideal for machine learning applications and skin reflectance analysis. Each one of these expression’s scans (combined data sets) took less than 0.28 of a second to capture.

We’re really excited about the potential of our new system and the quality of data that we can produce. We’re looking forward to working with clients on new and exciting projects and pushing the boundaries of what’s possible with 3D scanning and facial capture technology.

Edit 2022/12/22
Part of IR’s photometric process is based on the incredible research by Paul Debevec and the team at USC & ICT. In combination with our own software and hardware processes we use these licensed and patented technologies (license application pending 2023).

In the meantime we are commercially deploying a “Trilat” capture process in combination with image processing @ inciprocal.com

We hope to post more information related to the “Trilat” process soon.

Thanks for reading

Earlier in 2020 we spent some time testing out our new Ximea machine vision scanning system and our R&D processing pipeline. We’ve been so busy that only now have I had time to post some information about the workflow. We had the opportunity to scan Spencer C from BMA Models. This was our first chance to test out our full pipeline of scanning a subjects photometric skin details as well as a “motion scan” of their performance. We then planned to use their motion scanned performance to drive their photometric scan!

Today’s inspiration

[soundcloud url=”https://api.soundcloud.com/tracks/761143807″ params=”” width=” 100%” height=”166″ iframe=”true” /]

Firstly we acquired a basic set of globally lit diffuse albedo images, specular separated globally lit albedo images and a set of photometric normals of Spencer as outlined in a recent blog post.

Spencer’s likeness scan was then baked to a new low-resolution topology and the processed displacements maps were taken from the steered normals and applied to Spencer’s high-resolution ZBrush sculpt. These include wrinkles, creases and pore level details.

We then generated the additional appropriate real-time textures required to render Spencer correctly in Marmoset Toolbag 3.0 (this was pre Toolbag 4.0!).

Our S0L system was then used to capture and process a short performance sequence of Spencer with audio. We will use this RAW performance to drive the photometric likeness scan data.

Once the RAW data has been processed we can quality check it using a custom set of tools to render out the motion scanned frames and texture sequence in Marmoset Toolbag 3, as well as to relight the data in any lighting condition for further presentations.

We use markers on our scan subjects to help with 2D tracking later in the pipeline. This also helps with optical flow for the photometric textures as well as the diffuse albedo motion scan texture sequence. The examples below showcase an older marker layout.

We typically set our machine vision cameras to record at 60fps (60 scans a second). This allows us to capture much more nuanced in-between frames of the subjects. However we can shoot at 24fps, 25fps, 30fps, 48fps or even 50fps.

Our early R&D tests suffered with some LED flickering which can be fixed later on in post on the texture sequence. We also originally captured with a mixture of parallel and cross-polarized light. We now shoot fully cross-polarized.

We did some optical flow in Wrap4D from the static photometric scan to the motion scans mixed parallel and cross-polarized albedo texture to make sure the pore details were in sync between the motion scan session and the photometric session.

The guys at R3DS provide a very robust topology that’s built into Wrap4D. We then took the AEONX RAW motion scan data into R3DS’s Track and followed their tutorials here. After the data has been fully tracked (from a rigid sequence) it’s then wrapped and had optical-flow applied across the sequence, as well as other custom processes such as re-projection, further rigid tracking, skull blending etc. We can then add rigidly tracked digital eyes, teeth, hair details and an animated tongue into the sequence.
Rigid tracking is essential for eyes and teeth fitting. We will cover this in future tutorials.

Adding the additional details took minimal work if you have a stock set of parts to use, which we used from our triplegangers real-time models. We will be re-using the baked textures from the Photometric scan, so the previous captured markers would no longer be visible. However, you can remove the markers on the baked texture sequence using After Effects, if you need to use the motion scans albedo texture sequence, as you can see below.

We can then export and compile Spencer’s digital double performance into a real-time Alembic model to play back either in real-time or offline rendered in Marmoset Toolbag 3 or Unreal! We added some extra details like skin peach fuzz and stubble. We set up the materials and lighting to match some movie lighting reference!

The final results! We ran this test just to prove out that with our scanning system and services you can achieve unique looking realistic results on a decent budget. The process took about 1-2 days to complete, whilst learning how to use R3DS’s Wrap4D.

With the right equipment and tools the process is quite straight forward. We hope to offer our digital double processing services to the public very soon.

By using tailored photometric and captured motion scans of the original person, you will get much a more authentic looking result, less stock puppet and more nuanced.

The next step is to integrate static FACS multi-layered photometric details into the pipeline as a “machine learned” approached. More to come soon.

Thanks for reading

Today’s inspiration:

We would like to introduce you to our new motion scanning system.

The “AeonX Motion Scanning System”.

The word aeon /ˈiːɒn/, also spelled eon (in American English) and æon, originally meant “life”, “vital force” or “being”, “generation” or “a period of time”, though it tended to be translated as “age” in the sense of “ages”, “forever”, “timeless” or “for eternity”.

We have re-designed our in house custom motion scanning system. Now utilizing Ximea (www.ximea.com) PCI-E based cameras using the Sony Pregius sensor.

We can record at base speed 60fps at 12MP up to 120 FPS at 8MP. Our new system captures 35GB’s of RAW data every second. We use Global Shutter cameras that have precise hardware genlocked synchronization. Combined with separate synchronized audio recording.

We can now scan dynamic FACS sessions, which will typically record Terabytes of data at a time. We’ve built our own camera interface and storage system alongside Ximea. Using our own bespoke triggering hardware and software solution built from the ground up.

Since our first forey into motion scanning research back in 2009, we have progressed from testing with DSLR video recording all the way up to high resolution 20MP machine vision cameras.

3D Face Scanning – Custom Capture from Infinite-Realities 2010

Motion Scan Capture in Agisoft Photoscan from Infinite-Realities 2011

Agisoft Photoscan Videogrammetry from Infinite-Realities 2012

Multi-Shot Temporal Test. 10 from Infinite-Realities 2016

AeonX First Test from Infinite-Realities 2018 on Vimeo.

This video showcases our first ever Ximea test, only testing 16 synchronized cameras here, using constant illumination at 50fps, in 8bit mode, with audio synchronization. There was no solution like this available on the market. Everything from the camera control software to the audio recording, to the camera triggering, all had to be built from scratch.

To us, this system is a true marvel. When we first started building industry leading full body photogrammetry systems back in 2011, just being able to synchronize 16x cameras once every 15-30 seconds at 1/100th exposure was a huge undertaking. Now, we have to synchronize 48x cameras.. at 60 times a second. This required a custom built genlock triggering system.

We can synchronize our cameras at up to 1us and we are able to use exposure speeds exceeding 500us (1/2000th of a second). Utilizing LED synchronization (from ida-tronic.com) and fast exposure speeds we can eliminate motion blur and not blind our scan subjects!

Next Generation Motion Scanning from Infinite-Realities 2018 on Vimeo.

In late 2018 we were finally able to trigger and synchronize all of our 48x Ximea machine vision cameras. This was a real eureka moment.

Years of research, a bottomless pit of money and 1000’s of hours spent testing. With a great deal of help from Ximea, Bevis Tech Ltd and Ida-Tronic.

Each camera records at 750MB/s in 8bit mode or 1.5GB/s in 12bit (16bit) mode. In total all 48x cameras can stream to disk at 35GB/s in 8bit mode. Or 2.1TB/m. We can record performances for up to 5-10 minutes. Data storage will vary but a typical 1-minute recording can total around 2TB’s of RAW dng data.

This is all just one part of the puzzle. We then had to figure out how to organize, pre-process, store and reconstruct all this data as no current off the shelf pipeline existed.

Custom 4D Reconstruction Pipeline

In our early days of reconstruction testing, we utilized Agisoft Photoscan. Agisoft has excellent built-in timeline and processing tools (which we helped specify) but it lacked in some areas such as UV optimization and texture quality across time.

We have since moved over to Reality Capture with the help from the team at Capturing Reality we can now utilize CLI processing techniques using a set of custom-designed processing stages. Whilst the CLI tools set is rudimentary it does allow us to get the best detail possible from the application. We have since integrated Houdini into the mix using our own custom pipeline to break up processing into stages and to overcome inherent bugs in the CLI version that have yet to be fixed

Below you can see some of our very early processing tests, unoptimized RAW data.

RAW Scan Test from Infinite-Realities 2018 on Vimeo.

RAW Scan Test 2 from Infinite-Realities 2018 on Vimeo.

AeonX Scanning BR04 from Infinite-Realities on Vimeo.

AeonX Scanning BR09 from Infinite-Realities on Vimeo.

You can reach us at [email protected] for more information.