1. How much does rolling shutter really affect my shot?
There are two factors that contribute to how much rolling shutter is in a shot:
- The time difference between when the top line of the image was captured and when the bottom line was captured
- The amount of motion in the image
Yannix did an analysis of several thousand shots from over 20 different large Hollywood movies over the past year for which we supplied our clients with rolling shutter compensation as part of our normal matchmoving. We found that the average amount of rolling shutter was very close to 30%. This means that the time difference between the top scan line of frame 1 and the bottom scan line of frame 1 was about 30% of the time between frame 1 and frame 2 (typically 24fps, or ~0.0417secs).
A typical shot we encounter that was shot at 4k will have frames at which the features on the image are moving at more than 200 pixels per frame. 200 pixels per frame * 30% = 60 pixels of “time shift” between the top and bottom of the frame caused by the interaction of motion and rolling shutter.
On many of our projects, the VFX team can not accept a frame-to-frame slip of 1-2 pixels in magnitude, so failing to model the rolling shutter which can cause a 60 pixel warp of the plates between rapid movement frames and slowly moving ones will reduce the quality of your effect.
2. How does Yannix compensate for rolling shutter in a matchmove?
Yannix has developed proprietary software that can solve for the amount of rolling shutter in a given shot, and then correct for the artifacts caused by the rolling shutter when determining the camera and world reconstruction. This value will always be a constant over the course of a shot (unless there is a frame-rate change within the shot, or similar). Nonetheless, since the amount of the rolling shutter artifact is proportional to the amount of motion, and the motion is usually changing over the course of the shot, the actual amount of rolling shutter pixel distortion will be different on different frames. (You can see this on your mobile phone if you photograph a building while moving your phone rapidly side-to-side… sometimes the building will shear one way, and sometimes the other.)
Making sure your VFX pipeline can apply this rolling shutter compensation back onto rendered frames is very similar to lens distortion. Currently, Yannix supports delivery of the rolling shutter distortion in three different formats:
- GridWarpNode
- ST maps
- An open source Nuke node that Yannix has developed that clients can download and install
3. But can’t we just compensate for rolling shutter with other parameters like Z-rotation or lens distortion?
The total amount of slip caused by rolling shutter can be reduced by “cheating” other values. But this will usually introduce other artifacts that may be even less desirable than the rolling shutter artifact itself. For example, introducing an artificial animated Z-rotation into a rapid pan is unnatural, does not model the shearing of side-to-side rolling shutter artifacts, and will usually be problematic itself.
In order to compensate with a traditional (non-rolling shutter) lens distortion model, the distortion would need to be animated, and unnecessary animated lens distortion should normally be avoided in a VFX pipeline because it introduces a large risk that the lens distortion is actually just cheating a separate and unrelated issue with the matchmove setup. This can also introduce unnatural motion into the CG elements that will not match the plates.
By solving for a single, non-animated rolling shutter value for a given shot, and then applying a correction to every point on the CG rendered frame based on its own amount of motion, the VFX pipeline is accurately modeling the rolling shutter artifact that occurred in the camera, and is unlikely to introduce other problems.
Similarly to lens distortion compensation (which Yannix introduced into the matchmoving pipeline back in the late 1990’s), a VFX pipeline can benefit from standardizing rolling shutter compensation as just another step in the rendering and compositing process.
4. Why not solve every shot without rolling shutter compensation and turn it on only if there are rolling shutter related problems?
If a client prefers to avoid rolling shutter compensation in their pipeline when a matchmove of satisfactory quality can be found without it, such as when the amount of motion is small, or the effect is a bit more forgiving, Yannix will normally solve each shot both with and without rolling shutter compensation. If the effect on internal QC is not too noticeable, the client can either request that we deliver both versions, or just the “without rolling shutter compensation” version.
The biggest problem with this approach is about time delays. In VFX as in other industries, “time is money”. On average Yannix will deliver shots within about 3 working days of receiving the plates. (Assuming bid is approved rapidly.) 95% of these shots are approved on first delivery. Any kickback and discussion can easily add several more days if we consider the time for the client to review and generate notes, then Yannix needs to replicate the issue to confirm it wasn’t introduced at the clients facility (by using a modified BG plate, survey or adjusting camera parameters) and then any possible discussion that might become necessary if the issue is caused by inaccuracies in the survey or rolling shutter.
Any delays in delivering a full quality QC’d matchmove will delay the rest of the VFX pipeline, and our goal is to make sure that the matchmoving is as reliable as possible as quickly as possible so that there is more time for eventualities in other parts of the VFX pipeline.
Additionally, when a client asks us to attempt to address rolling shutter issues via other camera parameters, this often leads to several rounds of back-and-forth because those parameters are usually very imperfect solutions, and this introduces additional artifacts. This leads to even further delays and cost overruns.
5. Does a rolling-shutter compensation perfectly match the real rolling shutter in the camera?
Conceptually, matchmoving is about creating a perfect mathematical model of the world, and then reconstructing all the free parameters to best fit what is observed in the live-action plates. The simpler the model, the less perfect the fit. But more complex mathematical models become unwieldy and introduce a greater possibility that a mistake will be made and unintentional artifacts will be introduced.
So a CG camera will never “perfectly” match the real camera, but even very basic rolling shutter compensation can very accurately model the most significant effects of rolling shutter.
Yannix takes a very conservative approach to rolling shutter. Only a single static parameter is added to the entire shot when rolling shutter compensation is introduced (the scan time of the frame mentioned in section 1 above), minimizing the risk that the extra complexity will introduce new problems while trying to address rolling shutter.
6. I want to incorporate rolling shutter compensation into my render and compositing pipeline. How do I do it?
As mentioned in section 2 above, Yannix currently offers three different approaches for how a client can ingest and use the rolling shutter compensation that we can provide. On a modern workstation, the Yannix open source rolling shutter compensation Nuke node will normally add less than 0.2 seconds per frame to your compositing pipeline (for 4k images).
If you would like to further discuss the benefits of incorporating rolling shutter compensation into your pipeline, we are always available to support your needs. Please contact your Yannix representative, or email us at: info@yannix.com
Return to “What is Rolling Shutter? (Part One)”