One of the exciting possibilities in the AISOS space is the opportunity to combine technologies in new or novel ways. For example, combining RTI and photogrammetry may allow for 3d models with increased precision in surface details. Along these lines, we recently had the opportunity to do some work combining our GIGAMacro with our typical photogrammetry process.

This work was inspired by Dr. Hancher from our Department of English. He brought us some wooden printing blocks, which are made up of very, very fine surface carvings. His research interests include profiling the depth of the cuts. The marks are far too fine for our typical photogrammetry equipment. While they may be well-suited to RTI, the size of the blocks would mean that imaging with RTI would be very time consuming.

As we were pondering our options, one of our graduate researchers, Samantha Porter, pointed us to a paper she’d recently worked on which dealt with a similar situation.

By manually setting the GIGAMacro to image with a lot more overlap than is typical (we ran at a 66% overlap), and using a level of magnification which fully reveals the subtle surface details we were interested in, we were able to capture images well suited to photogrammetry. This process generates a substantial amount of data (a small wooden block consisted of more than 400 images), but it’s still manageable using our normal photogrammetry tools (Agisoft Photoscan).

After approximately 8 hours of processing, the results are impressive. Even the most subtle details are revealed in the mesh (the mesh seen below has been simplified for display in the browser, and has had its texture removed to better show the surface details). Because the high-overlap images can still be stitched using the traditional GIGAMacro toolchain, we can also generate high resolution 2d images for comparison.

We’re excited to continue to refine this technique, to increase the performance and the accuracy.

As we’ve previously discussed, the Gigamacro works by taking many (many) photos of an object, with slight offsets. All of those photos need to then be combined to give you a big, beautiful gigapixel image. That process is accomplished in two steps.

First, all of the images taken at different heights need to be combined into a single in-focus image per X-Y position. This is done with focus-stacking software, like Zerene Stacker or Helicon. After collapsing these “stacks,” all of the positions need to be stitched together into a single image.

On its surface, this might seem like a pretty simple task. After all, we’ve got a precisely aligned grid, with fixed camera settings. However, there are a number of factors that complicate this.

First off, nothing about this system is “perfect” in an absolute sense. Each lens has slightly different characteristics from side to side and top to bottom. No matter how hard we try, the flashes won’t be positioned in exactly the same place on each side, and likely won’t fire with exactly the same brightness. The object may move ever so slightly due to vibrations from the unit or the building. And, while very precise, the Gigamacro itself may not move precisely the same amount each time. Keep in mind that, at the scale we’re operating at (even with a fairly wide lens), each pixel of the image represents less than a micron. If we were to blindly stitch a grid of images, even a misalignment as small as one micron would be noticeable.

To solve this, the Gigamacro utilizes commercial panorama stitching software – primarily Autopano Giga. Stitching software works by identifying similarities between images, and then calculating the necessary warping and movement to align those images. For those interested in the technical aspects of this process, we recommend reading Automatic Panoramic Image Stitching using Invariant Features by Matthew Brown and David Lowe. In addition to matching photos precisely, these tools are able to blend images so that lighting and color differences are removed, using techniques like seam carving.

While this type of software works well in most cases, there are some limitations. All off-the-shelf stitching software currently on the market is intended for traditional panoramas – a photographer stands in one place and rotates, taking many overlapping photos. This means they assume a single nodal point – the camera doesn’t translate in space. The Gigamacro is doing the opposite – the camera doesn’t rotate, but instead translates over X and Y.

Because the software is assuming camera rotation, it automatically applies different types of distortion to attempt to make the image look “right.” In this case though, right is wrong. In addition, the software assumes we’re holding the camera by hand, and thus that the camera might wobble a bit. In reality, our camera isn’t rotating around the Z axis at all.

Typically, we fool the panorma software by telling it that we were using a very, very (impossibly) long zoom lens when taking the photos. This makes it think that each photo is an impossibly small slice of the panorama, and thus the distoration is imperceptible.

However, Dr. Griffin from our Department of Geography, Environment & Society presented us with some challenging wood core samples. These samples are very long, and very narrow. Even at a relatively high level of zoom, they can fit within a single frame along the Y axis. Essentially, we end with a single, long row of images.

This arrangement presented a challenge to the commercial stitching software. With a single row of images, any incorrect rotation applied to one image will compound in the next image, increasing the error. In addition, the slight distortion from the software attempting to correct what it thinks is spherical distortion means the images end up slightly curved. We were getting results with wild shapes, none of them correlating to reality.

Through more fiddling, and with help from the Gigamacro team, we were able to establish a workflow that mostly solved the problem. By combing Autopano Giga with PTGui, another stitching tool, we were able to dial out the incorrect rotation values and get decently accurate-looking samples. However, the intention with these samples is to use them for very precise measurements, and we were unconvinced that we had removed enough error from the system.

As mentioned earlier, the problem appears, on its face, to be relatively simple. That got us to thinking – could we develop a custom stitching solution for a reduced problem set like this?

The challenging part of this problem is determining the overlap between images. As noted, it’s not exactly the same between images, so some form of pattern recognition is necessary. Fortunately, the open source OpenCV project implements many of the common pattern matching algorithms. Even more fortunately, many other people have implemented traditional image stitching applications using OpenCV, providing a good reference. The result is LinearStitch, a simple python image stitcher designed for a single horizontal row of images created with camera translation.

LinearStitch uses the SIFT algorithm to identify similarities between images, then uses those points to compute X and Y translation to match the images as closely as possible without adding any distortion. You might notice we’re translating on both X and Y. We haven’t yet fully identified why we get a slight (2-3micron) Y axis drift between some images. We’re attempting to identify the cause now.

At this point, LinearStitch isn’t packaged up as a point-and-click install, but if you’re a bit familiar with python and installing dependencies, you should be able to get it running. It uses Python3 and OpenCV3. Many thanks to David Olsen for all his assistance on this project.

We’ve had the Gigamacro and the photogrammetry capture station up and running for about a week now. While both technologies are relatively straightforward in terms of the technology, it’s clear that both benefit from a lot of artistry to get the most out of them.

The Gigamacro is conceptually very straightforward. It’s just a camera that takes many close-up photos of an object, and then combines all those photos into a single high resolution output. The camera uses a fixed lens (rather than a zoom), and we have a variety of lenses with different levels of magnification. Two of the challenging parts of using the Gigamacro are focus and lighting. At very close distances, with the types of lenses we’re using, the “depth of field” is very narrow. Depth of field is something we’re familiar with from traditional photography – when you take a picture of a person, and the background is nicely blurred, that’s due to the depth of field. If everything in the photo is in focus, we say that it has a very wide depth of field.

With the Gigamacro, and using a high magnification lens, the depth of field is typically on the order of a few hundredths of a millimeter. The Gigamacro solves this by taking many photos at different distances from the object. Each of these photos can then be combined (“depth stacked”) into a single photo in which everything is in focus. Even on a surface that appears perfectly flat, we’re finding that a few different heights are necessary. With a more organic object, it’s not unusual to need to capture 40 or 50 different heights. Not only does this greatly increase the number of photos needed (we’re currently working on a butterfly consisting of 24,000 photos) but it increases the post-processing time necessary. All of this means we’re greatly rewarded for carefully positioning and preparing objects to minimize height variation when possible.

Another issue is light. The Gigamacro has two adjustable flashes. These mean there’s plenty of light available. At very close distances though, it’s important not to cast shadows or overexpose areas. We’re starting to get a better understanding of how to adjust the lighting to deliver quality results without losing detail, but we’re still learning. In addition, the package we have includes some filters for cross-polarizing light, which in theory will allow us to capture very shiny surfaces without reflections.

So, what’s it all look like? We’re still working on building a sample gallery, but below is one example. This is a small fish fossil, captured with a 100mm lens. The original object is approximately 3 inches across. In terms of the gigamacro, this is a very low resolution image – only approximately 400 megapixels. We’re finding that this technology is much more impressive when you’ve seen the physical object in person. Only then do you realize the scale of the resolution. We’ll be sharing more samples as we get more experience. Just click to zoom. And zoom. And zoom.

Photogrammetry

Photogrammetry is the other main technology we’re working with at this phase. Our new photogrammetry turntable is working well, and we’re continuing to explore different lighting setups and backdrop options for the space.

Because we’ve got a processing workstation near the photogrammetry station, we’re able to stream photos directly from the camera to the computer. There’s no need to manually transfer files. This, combined with the automation of the turntable, means we can do a basic photogrammetry pass on an object very quickly, then make adjustments and try again.

Below is one of our first finished objects, a small plastic dinosaur. This object is far from perfect – in particular, the tail needs additional cleanup, and there’s some “noise” around the base. This is a combination of two passes, capturing the top and the bottom of the dinosaur. It’s made of 159 individual photos, all processed with Agisoft Photoscan Pro. We’re excited to compare Photoscan with Autodesk Remake, but a recent Windows update broke Remake. Hopefully they’ll get that fixed soon.

We made good progress in the AISOS space today. The photogrammetry (and later, RTI) camera position is in place. We were lucky to inherit a nice copy stand that was already in the space, which makes camera positioning really easy. We were even able to capture our first object, albeit very roughly (the lightning needs plenty of adjustment).

Later in the day, we got the call that our GigaMacro had arrived. Anything that comes in a massive wooden crate is bound to be exciting.

We don’t have our final work surface for the GigaMacro yet, but we did some initial assembly and testing. Everything seems to be working as expected. It’s definitely going to have a learning curve as we get familiar with all the variables that can be controlled on the GigaMacro. For now, you can watch it wiggle.

The AISOS space has now been cleaned, painted, and the ethernet jacks are in. Most of our equipment is on-site, with the exception of the Gigamacro which should be arriving next week.

We’re going to begin experimenting with different arrangements for the space. We’ve also got a few more pieces of furniture to move in.

We’re really excited to be moving forward with AISOS. We’re currently in the process of preparing our space, and ordering all of our equipment. We’re hoping to have most of the equipment installed and ready to go by the end of August.

If you’re curious to learn more, send us an email. Otherwise, look for a tour soon!