CMOS Rolling Shutter Removal - Plugin Shootout!

I'm getting away with whip pans like these.

Or just about, anyway.

That said, I'm not going to bore you with an explanation of what rolling shutter is. Chances are that - before you reached this blog - you already went through the torture of a few hundred posts and videos of everyone else demonstrating rolling shutter - without the slightest suggestion of a good CMOS skew and jello removal tool.

You can relax now, for I'm going to show you a comparison between the five filters that are available for doing just that, and the most professional one of the lot.

Better yet, the best one isn't even the most expensive.

Following is our list of candidates:

1. The DeShaker plugin for VirtualDub:
• Free; requires that you convert your footage to AVI before importing. 
2. Adobe's After Effects' Warp Stabilizer, available since CS5.5:
• Requires purchase of one of the Adobe suites. Not necessarily cheap.
• Note: CS6 is equipped with a dedicated rolling shutter corrector, but I did not have time to download CS6 to try it.
3. ProDAD's Mercalli PRO Version 2
• $249 from the ProDAD website
• Sony Vegas - since version 10 - contains a simplified version of Mercalli; we'll cover only the standalone.
4. New Blue FX's Video Essentials III plugin, which contains a rolling shutter removal tool.
• Works with Adobe After Effects, AVID Media Composer, and Sony Vegas, amongst others. $130, plus the cost of your editor.
5. The Foundry's Rollingshutter plugin for Nuke and After Effects
• A cool $500.

Do note that I'm taking this from the point of view of PC's, though MAC users have a rolling shutter removal tool in Apple's Final Cut 10, and the Foundry's plugin is also MAC compatible.

Seeing as there is no better way to see the results than a video, I prepared three rolling shutter examples and ran them through each of the five filters:

 

Though not covered in the video, program usability presents a concern as well; after all, a timely workflow is essential to smooth editing.

Conclusions:

• DeShaker is an absolute nightmare to use. Besides having to convert footage to AVI (I used Pazera MOV to AVI to do so - it was quicker than Adobe Media Encoder) before even importing it, setting the plugin up in VirtualDub is a hassle, and it takes ages to analyze the footage. You'll wind up with some pretty big AVI files to clean off your drive too unless you install and compress with the x264vfw plugin.
• Warp Stabilizer, along with the New Blue FX and Foundry plugins run the same as most other effect plugins for After Effects. New Blue FX had a tendency of crashing AE at times, while The Foundry's plugin can become a long and drawn-out process.
• Mercalli is ridiculously fast, and arguably the most user intuitive. It almost looks too simplistic and mainstream to be a professional tool, but - as proven by the footage tests - it is superior to everything else as a general-purpose tool. 
• At best, the only further improvement would be to correct individual extreme skew issues with New Blue FX before importing into Mercalli. With certain footage, this workflow could remove rolling shutter skew entirely.

One last option remains for CMOS skew removal (exclusively - this method will not correct for jello/jitter), if you do not mind a bit of DIY: Layer seven or eight feathered copies of the problematic footage over each other, with each successive clip (from top to bottom) playing progressively slower than the first - thereby correcting for the top-to-bottom scan delay in most CMOS sensors.

Mike Huetz on Vimeo has a brilliant video tutorial of this method: http://vimeo.com/10130633

Regardless, a powerful filter does not negate that one can ignore CMOS rolling shutter when shooting, expecting that you can fix everything in post. That's not a reasonable though - you simply cannot walk into a shoot and expect to get away with it.

If absolute perfection is what you need, the Black Magic Production Camera 4K - with global shutter - should be your next purchase, rather than a rolling shutter corrector.

In fact, I fully intend to become the Man with the BMPC 4K someday, but that's the future.

Where did I leave my DeLorean?

No monetary compensation was provided for the mention of any of the products listed. All were tested during product trial periods or were purchased by myself.

Aperture Flicker: Why you should have a manual aperture zoom lens...

...and why electronic, variable aperture zoom lenses are a poor choice for DSLR filmmaking:

Aperture flicker is a phenomenon of zoom lenses with variable apertures at the open end. For the purposes of this explanation, let us consider the following Canon EF 28-135mm zoom lens - it will run open as low as f/3.5 when at 28mm, but will close to f/5.6 when the focal length increases to 135mm.

As the lens has an electrically controlled aperture built into the body, the lens will automatically stop down from f/3.5 to f/5.6 when zooming, causing an abrupt reduction in light as the iris is closed in individual clicks/stops.

It is the same look you'd get if you operated the iris/aperture of a manual lens that had not been de-clicked for film/cine use.

Now, at this point, you're probably saying: "Wait - what if I simply stopped down to f/5.6 for the entire shot? That makes perfect sense, doesn't it?" It certainly does, and you'd be dead wrong, despite what many DSLR photographers (as opposed to videographers) would like you to believe.

In short:  

Even if you stop down to an f-stop that works across the entire focal length of the zoom lens, IN MANUAL MODE, the electric aperture will flicker anyway.

The problem lies in the hard-coded programming of DSLR cameras, which checks at each of the critical stop-down points in the focal range to see if the aperture needs to be stopped down when you zoom, activating the iris mechanism for a split second, thereby causing the momentary darkening of the image. It will do so even if you already stopped down the lens to prevent the system from stopping it down for you.

That's right - so long as you have an variable aperture zoom lens with electronic iris/aperture control (more or less a Canon EOS and M43 issue), you cannot use the zoom on your lens without getting completely uneven light. 

For those of you who want to see this as a video, following is a direct comparison of an all-electronic Canon EF 28-135mm zoom lens  - with a variable f/3.5-5.6 max aperture - against a 1980's Tamron 28-70mm zoom. Just to make the comparison more stark, the Tamron is also a variable maximum aperture lens (f/3.5-4.5, specifically), but it adjusts mechanically - and therefore does not distract from the image.

 

Obviously, aperture flicker is not a problem for still photography, but it proves to be a severe issue for videos.
 
DSLR filmmakers are left with three options to get around aperture flicker:

1. Use zoom lenses made for your existing mount that will run a constant open f-stop across the entire focal range
• Some common zoom lenses are manufactured as described (the EF-mount Canon 24-70mm f/2.8 L - both the original and the II - and Panasonic's 35-100mm f/2.8 for M43 come to mind), but this may needlessly restrict your lens options.
• Note that if you are shooting telephoto-length lenses, using a modern, constant aperture lens is your ONLY option, due to the requirement of image stabilization and a fixed zoom ring. Admittedly, this solution is pricy, I cannot think of a better excuse to justify a 70-200mm Canon f/2.8 L IS to your significant other.
2. Use a lens made for your existing mount that has manual iris control.
• To my knowledge - short of Cinematics' customized (and pricey) cine lenses - non-existent at the prosumer level. Rokinon/Samyang's budget cinema lenses for EF mount are all primes, as are Noktor's SLR Magic series for Micro 4/3.
3. Use an older manual zoom lens with an adapter, preferably de-clicked.

Redrock Micro's LiveLens MFT

But there is a fourth way too:

Granted, this option only exists for M43 users who wish to use EF-mount lenses, but it is an option at any rate: Redrock Micro's M43-to-EF LiveLens MFT, shown at right.

Upon request, Redrock was kind enough to report that the LiveLens controller does not cause aperture flicker when used with a variable aperture lens. Specifically, the LiveLens was tested with a Canon 20-35mm f/3.5-4.5; when set at at f/8, the lens exhibited no aperture flicker when zoomed and retracted.

One last note: Obviously, photographers don't need to worry about this problem, but it hasn't prevented a number of discussions about aperture flicker to appear on photography forums. Be forewarned that a good deal of the advice written on these forums - though well-meaning - are often misleading, or do not apply to DSLR filmmaking.

Kurt K. - "Man With the 5D" 

No monetary compensation was provided for the mention of the LiveLens MFT for this article. LiveLens MFT image and data was provided by Redrock Micro upon my request.