## Monday, March 23, 2015

### Turning a Machine Vision Camera into a Digital Cinema Camera

(with a title that pretentious it has to be good, right?)

Intro

Recently Sony released the IMX174 sensor, a 16mm-format global shutter CMOS sensor good for 150FPS at 1920x1200. The sensor is current available in several machine vision cameras for around $1000, including the Point Grey Grasshopper3 and the Basler Ace. I had been looking for a good high-speed video camera lately that could double as a day-to-day raw camera for general videography; while the Redlake was perfectly good camera, its absurd light requirements, short record times, and general bulk limited it to special occasions. Inspired by Shane's recent success with using a Grasshopper3 on a Freefly rig for general video, I decided to give it a go using a Basler camera and my own spin on what I thought the UI should look like. The Camera I managed to hunt the camera in question (acA1920-155uc) in stock at Graftek; it was overnighted to me and the next day I had it in hand. The first thing that struck me was how small the head was - at 29mm on a side the camera was barely larger than a dollar coin. The next thing that struck me was how bad the stock viewer was - AOI didn't work, and recording dumped debayered BMP files frame-by-frame to the disk. Basler promised me that AOI would be implemented soon, but didn't have a good answer for the recording issue ("use Labview" was not a valid answer!) But as I often say, "its just code"... First Light Getting RAW images frame-by-frame out of the camera was easy - there was a Pylon SDK example for grabbing raw framebuffer data. Throughout this project, Irfanview proved to be invaluable for viewing and debayering raw frame data. It was fairly straightforward to write the raw frames frame-by-frame to disk, but that was good for under 60FPS due to the overhead of file creation. The logical next step was to buffer the images to an in-memory buffer and then flush the buffer several hundred MB at a time to disk - using this strategy I was able to reliably grab at maximum frame rate... ...until about 5000 frames, at which point disk write speeds would plummet to <150MB/s and frames would start dropping like crazy. I spent a day trying to fix this, thinking it was a resource handling issue on my end, until I realized that the SSD on my development machine was nearly full. Most consumer SSD's have terrible write speeds towards the end of their capacities, as they run out of free blocks and have to start erasing and rewriting entire blocks for even small amounts of data. Deleting some files and trimming the drive seemed to fix the issue, and I could record continuously at maximum frame rate (with frames cropped from 1200p down to 1080p) so long as background processes were closed. Getting a Preview After much derping with GDI bitmaps and DirectX, I remembered that SDL existed and, as of version 2.0, was hardware-accelerated. A rough preview using terrible nearest-neighbor debayering was easy. Unfortunately, while the chunk-based disk writing scheme significantly boosted storage performance, it also had the side effect of freezing preview during the disk writes. This was OK for low data rates, but not for high frame rates where the disk would be mostly busy.The natural solution was to run the framebuffer update in a separate thread, which was easier said then done - Pylon ostensibly has callback functions that can be attached to its internal grab thread, but enabling callbacks seemed to disable Pylon's internal buffer update. The solution was to write a wrapper around the Pylon InstantCamera object that ran in an SDL_thread and updated an external buffer. Finding a Host Computer I wasn't particularly keen on paying$1000 for a Surface Pro 3 and the firesale on obsolete SP1's had largely ended, but fortunately the Dell Venue 11 Pro uses an M.2 SSD in the Core i3 and i5 versions and costs about \$300 refurb. The stock SSD is somewhat lacking - it tops out at 300MB/s and needs a little bit of warming up in the form of a few hundred frames written before it can sustain this speed, but was OK to start with.

Also worth noting at this point I ran into random dropped frames, which turned out to be a crappy USB cable; unlike USB 2.0, USB 3.0 actually needs a good cable to sustain maximum transfer rate.

The horrible nearest-neighbor preview also looked pretty awful on the Venue's low-DPI screen (1080p on 11" versus 1600p on 13" on the laptop I had written the code on), so it was time to write a quick-'n-dirty software bilinear debayering function. It runs a bit hot (~40% CPU on the Venue) but seems good enough for 30FPS preview. There were also considerable tearing problems, which turned out to be a combination of thread contention in the buffer update code and a lack of Vsync during the screen update.

Better Post-processing

At this point I could grab raw ADC data to disk, but that wasn't too useful for producing viewable videos. I wrote a quick player that used bilinear debayering to preview the videos, and used Windows file associations to avoid having to write an actual GUI for it (this also made it more touch-friendly; also, cool trick: the Windows "Open With" menu option passes the file name as argv[1] to the program it calls). The 's' key in the player cycles between LUTs (currently there are linear, S-Log, Rec709, and an curve of my own creation that I thought "looked good"), 'd' dumps .RAW files for Irfanview or dcraw, and 'b' dumps bilinear debayered BMP's.

I also wanted better debayering without actually having to write a better debayer-er. After scrolling through more lines of pure global-variable based C than I ever wanted to (its somewhat horrific that most of the worlds' raw photo handling stems from that mess), I figured out how to add support for new formats to dcraw. For metadata-less files, dcraw picks the file type based on file size.

Sensor performance

The chart says it all:

 Stops of DR versus stops of gain
The sensor has a base ISO of around 80.The plot above shows stops of DR versus gain in 12-bit mode (which is good for 78 FPS). Performance at base ISO is excellent; performance at higher ISOs is terrible enough to the point where digitally applying gain in post is probably better for most applications. At high gain there is severe banding noise originating from a yet unlocated source.

Download the package here. You will need the Visual C++ runtime DLL's, Basler's Pylon 4 runtime (you'll probably want to uninstall Pylon 5), an acA1920-155uc camera, a USB 3.0 port, and a reasonably fast SSD. Use Pylon Viewer to set the bit depth and use either grab_12p_fs or grab_8_fs to record files. To use the player, associate .aca files with play_12p.exe and .ac8 files with play_8.exe. Inside the player, 'd' dumps RAW frames, 'b' dumps BMP's, and 's' cycles between curves.

1. In 12-bit mode, does the Basler put the 8 MSB's of adjacent pixels in Byte N and Byte N+2 and the four LSBs together in Byte N+1? The Point Grey does that for some odd reason and I'm wondering if it's a sensor thing or a camera/implementation thing.

2. The Basler does a similar thing but not quite like the Point Grey's, if three adjacent bytes are AB CD EF the mosf significant bytes of the two pixels are DA and EF.

3. Hi! Just testing your package! Works perfectly :-)
Just a question - which settings do you use for Irfanview?
I have the same acA1920-155uc and only get bw e.g. error in Irfanview...
Lightroom does not recognize the raw file, perhaps dere is another debayer tool?

The new basler PGI feature firmware looks nice but the drawdown is the reduced bitrate and so less frames per second...

kind regards fron Graz,
Robert

1. Whoa impressive, it actually works on other people's computers!

I believe the Irfanview setting is something like 12-bit Bayer, you will have to go cycle through the settings in that category until you find the one that works since I don't have any Basler raw's on this computer at the moment.

2. Also, I just uploaded a new recorder with support for higher framerates at lower resolutions (180 at 720P, 300 at 480P); the new package comes with a patched binary of dcraw (run it in the command line with dcraw -4 -q 3 ) that should give you the utmost image quality (dcraw is balls slow however, so be patient).

3. Do you mean utmost quality just for viewing or is it also for exporting the BMP files? I am looking for the best possible picture quality ;-)

4. It exports the file as a 16-bit TIFF, which can be manipulated with your favorite raw photo editor.

5. Also FYI, the settings you'll want to use for Irfanview are 8BPP, Bayer Pattern RG

6. Perfectly! Everything is running well with this settings:
Irfanview 1920x1080 (! not 1200), 8BPP, vertical flip, Bayer pattern RG.
Dcraw command line: dcraw -4 -q 3 -T filename.raw

Do you think to implement the new Basler pylon 5.0.0.6110 runtime in the future?

best Robert

7. Probably going to stick with Pylon 4 for now, the recording end of this project is more or less done - the magic will be in the postprocessing tools to convert the ac8 files into some reasonable format (tiff? CinemaDNG?)

4. Wow that are great news! I wish i could have had the opportunity yesterday - we made some shootings of a horse hoof - unfortunately "only" with 155fps at full resolution. If anybody is interested for experimenting with an exported raw file (exported with the version 1.0 of the raw recorder) here you go: https://www.dropbox.com/s/2xwle287fndhe5x/DSC1465316576_146.raw?dl=0

5. Hi
Experimenting the package today with no luck. I use a acA1920-40uc. Copied the Pylon 4.2.2 dll files (the Pylon 4.X.X redistributable packages are no more downloadable, so I installed the 4.2.2 full SDK onto a VM and pick the needed DLL. I do this to avoid conflicts with the Pylon 5 SDK which is installed on the same computer).

Starting grab8 or grab12p opens briefly a console-like window then crashes.

By the way our computer runs Under Win10 x64. Any clues ?

6. OK. I had to remove all the bits of the Pylon 5.X.X installation and install 4.2.2 instead in order to make it work.
It runs fine now. Some flaws on the GUI side : The "minus.bmp" does not show the minus sign it is a flat square, and checking "disable display scaling on high DPI settings" in the compatibility mode helps a lot on high-res screens.

1. never had a problem with the icons not showing up, but good to know it doesn't play well with Pylon 5, and added a note to the post.
I don't run scaling on my laptop display, I'm curious, what does it do with scaling on?