by guest contributor David Brady, Duke University
For the first time in the history of the America's Cup, this year's races in San Francisco were held in a natural amphitheater observed along the shore. Spectators along the waterfront got real-time race data on smartphone apps while television broadcasts overlaid boat track, speed and separation along with wind and tide direction. Along with revolutionary boats and dramatic narratives, the races of the 34th America's Cup provided an excellent opportunity for computationally enhanced and interactive media. Such a dramatic event taking place on such a large scale was a natural target for panoramic photography.
For the first time in the history of the America's Cup, this year's races in San Francisco were held in a natural amphitheater observed along the shore. Spectators along the waterfront got real-time race data on smartphone apps while television broadcasts overlaid boat track, speed and separation along with wind and tide direction. Along with revolutionary boats and dramatic narratives, the races of the 34th America's Cup provided an excellent opportunity for computationally enhanced and interactive media. Such a dramatic event taking place on such a large scale was a natural target for panoramic photography.
Duke University’s Imaging and Spectroscopy Program (DISP), with support from the Defense Advanced Research Projects Agency, builds compact, wide field of view, high-resolution cameras ideal for the capture of events such as the America’s Cup. In contrast with traditional, scanned panoramas, DISP's Aware cameras take instantaneous snapshots of the full field of view. DISP currently has two such cameras, named Blue Devil and Wild Cat based on the Aware 2 design. Blue Devil uses 96 14-megapixel micro-cameras to produce 700 megapixel images over a 120 degree by 50 degree field of view. Wild Cat uses 30 micro-cameras to produce 200-250 megapixel images. DISP, and DISP spinoff Aqueti, have numerous next generation panoramic cameras in fabrication, including Triton, a 3-gigapixel snapshot camera, and the Aqueti qG, a tripod-mountable, 250 megapixel camera.
With support from Gigapan, DISP had already used Aware 2 cameras to capture and distribute high definition snapshots. We used Gigapan Tag with Blue Devil to produce a taggable photo of Duke's 2013 graduation ceremony. Between Blue Devil and Wild Cat, we captured over 400 snapshots at that event. Unfortunately, fully automated panorama production is not yet producing satisfactory results. Most DISP images posted on Gigapan are optimized using Kolor's Autopano Giga, which we used with Wild Cat to produce high definition snapshots at recent Durham Bulls and Duke Blue Devil games. The Aware cameras are computational imagers, ultimately requiring full camera operating systems with advanced exposure and focus control, novel image data formats, and sophisticated image formation and display strategies. As we continue to develop Aware software and hardware, event capture has been helpful in defining essential specifications and operational challenges and opportunities.
After months of testing and activity, and rather than waiting for Triton or qG, DISP went to the America's Cup with the camera on hand. Although somewhat challenging with Wild Cat and Blue Devil as prototype cameras in large boxes, races 6 through 10 of the 34th America's Cup were the most exciting shooting campaign to date for the DISP Aware team.
Shipped from North Carolina to San Francisco, DISP rented a big truck to haul Wild Cat around. Here are DISP's staff photographer, Jack Anderson, and the Aware project manager, Steve Feller, with Wild Cat on the truck:
The America's Cup Event Authority generously supported DISP with media passes and site support for the campaign. While we were familiar with the AC72's and had seen them cruising the bay on television, nothing could match the seeing them from the staging area off of Pier 23:
We took our first snapshots of these boats up close on the morning of Thursday, September 10:
Ready to Rumble
Later that afternoon, with the assistance of America's Cup SUVs and golf carts, we staged Wild Cat at the Golden Gate Yacht Club and observed races 6 and 7 looking toward the Golden Gate Bridge:
Race 7 of the 34th America's Cup
Friday, September 13 was not a racing day, so we spent time wandering around America's Cup Park and shooting self portraits. We were surprised to find an interesting snapshot in this one:
Bare Foot Guy Working High Above Athena
We deployed the truck in the tailgate region of America’s Cup Village at Marina Green for races 8 and 9 on Saturday, September 14:
This vantage point gave us a clear view of the race course over the yacht clubs and across the spit that we affectionately named Kiwi Point:
Kiwi Point
The image across Kiwi point was our favorite of the campaign, coming in race 8. Race 9 was cancelled midway through after winds exceeded threshold. We got a reasonable shot of the crowd at Marina Green leaving when the cancellation was announced.
Leaving the Cup After the Race
We loved shooting the races at the Golden Gate Yacht Club and at Marina Green, but our friends at the America's Cup let us know that they had enough pictures of boats. We feel, of course, that our shots of the AC72's in the context of the full fleet in the Bay and the crowd along the shore give unique opportunities to explore the full scene but we agreed that on Sunday, September 15 we would shoot pictures of the crowd at Marina Green rather than the race. Once we had deployed Wild Cat on Kiwi Point we had trouble explaining to spectators why we chose to point the camera the wrong way. We decorated Wild Cat with both flags to remain politically unbiased.
Leaving the Cup After the Race
We loved shooting the races at the Golden Gate Yacht Club and at Marina Green, but our friends at the America's Cup let us know that they had enough pictures of boats. We feel, of course, that our shots of the AC72's in the context of the full fleet in the Bay and the crowd along the shore give unique opportunities to explore the full scene but we agreed that on Sunday, September 15 we would shoot pictures of the crowd at Marina Green rather than the race. Once we had deployed Wild Cat on Kiwi Point we had trouble explaining to spectators why we chose to point the camera the wrong way. We decorated Wild Cat with both flags to remain politically unbiased.
The range from Kiwi Point to the stands is a bit far for Wild Cat’s 115mm focal length equivalent, but we hope that the folks on the shore can at least recognize themselves:
America's Cup Crowd
We also experimented with Gigapan Tag with these images, as seen here.
Our goal at the America's Cup was to capture as many images as we could and put them online as quickly as possible. With the use of the America's Cup Media Center for image processing and the efficient Gigapan Upload utility, we were able to capture several hundred images each day, scan for our favorites and post several of our choosing. In the end, we posted 26 images. Each image requires 15-30 minutes of processing time, with much still to learn about optimal processing.
While we are developing real-time processing systems, images posted from the America's Cup use Matlab image processing scripts and Autopano Giga for stitching. The processing scripts are still quite rudimentary; the posted images are formed from nearly raw sensor data. With a goal of posting images within ten minutes, we were unable to achieve as much quality as we can with further processing. We learned a lot about how to make our images better. In analyzing the Kiwi Point image, for example, we found that edge aware demosaicing (below right) outperforms bilinear demosiacing (below left):
We also learned that we could substantially improve our images with more advanced denoising and color processing, as shown in this comparison of one of the Marina Green crowd shots after Photoshop processing:
While improved image processing is an easy fix for Aware images, the most significant artifacts in our America's Cup images arise from differences in exposure and focus across the micro-camera array. The capacity to independently manage exposure and focus is, of course, a central advantage of the multiscale architecture. Management of this advantage is, conversely, a serious challenge. With more time to control contributions from diverse micro-cameras, we are providing (below) a better focused version of the Kiwi Point data. The colors are also richer, although nonuniformity in the stitching of the sky shows why we cropped it in the fast stitch version.
Kiwi Point Once Again
We thank the America's Cup and Gigapan for helping us get through this experiment. While our images are not perfect, they are the only zoomable, multi-scale images that will ever exist for this America's Cup. We look forward to taking the lessons learned from this experience in building gigapixel snapshots of future critical moments in time.
