Parsing and plotting OMNIC Specta SPA files with R and PHP

This is a quick “howto” post to describe how to parse OMNIC Specta SPA files, in case anyone goes a-google’n for a similar solution in the future.

SPA files consist of some metadata, along with the data as little endian float32. The files contain a basic manifest right near the start, including the offset and runlength for the data. The start offset is at byte 386 (two byte integer), and the run length is at 390 (another two byte int). The actual data is strictly made up of the little endian floats – no start and stop, no control characters.

These files are pretty easy to parse and plot, at least to get a simple display. Here’s some R code to read and plot an SPA:

pathToSource <- "fill_in_your_path"; = file(pathToSource, "rb");

# Read the start offset
seek(, 386, origin="start");
startOffset <- readBin(, "int", n=1, size=2);
# Read the length
seek(, 390, origin="start");
readLength <- readBin(, "int", n=1, size=2);

# seek to the start
seek(, startOffset, origin="start");

# we'll read four byte chunks
floatCount <- readLength/4

# read all our floats
floatData <- c(readBin(,"double",floatCount, size=4))

floatDataFrame <-
p.plot <- ggplot(data = floatDataFrame,aes(x=ID, y=floatData))
p.plot + geom_line() + theme_bw()

In my particular case, I need to plot them from PHP, and already have a pipeline that shells out to gnuplot to plot other types of data. So, in case it’s helpful to anyone, here’s the same plotting in PHP.


function generatePlotForSPA($source, $targetFile) {

    $sourceFile = fopen($source, "rb");

    fseek($sourceFile, 386);
    $targetOffset = current(unpack("v", fread($sourceFile, 2)));
    if($targetOffset > filesize($source)) {
        return false;
    fseek($sourceFile, 390);
    $dataLength = current(unpack("v", fread($sourceFile, 2)));
    if($dataLength + $targetOffset > filesize($source)) {
        return false;

    fseek($sourceFile, $targetOffset);

    $rawData = fread($sourceFile, $dataLength);
    $rawDataOutputPath = $source . "_raw_data";
    $outputFile = fopen($rawDataOutputPath, "w");
    fwrite($outputFile, $rawData);
    $gnuScript = "set terminal png size {width},{height};
        set output '{output}';

        unset key;
        unset border;

    plot '<cat' binary filetype=bin format='%float32' endian=little array=1:0 with lines lt rgb 'black';";

    $targetScript = str_replace("{output}", $targetFile, $gnuScript);
    $targetScript = str_replace("{width}", 500, $targetScript);
    $targetScript = str_replace("{height}", 400, $targetScript);
    $gnuPath = "gnuplot";
    $outputScript = "cat \"" . $rawDataOutputPath . "\" | " . $gnuPath . " -e \"" . $targetScript . "\"";
    if(!file_exists($targetFile)) {
        return false;
    return true;

Transcoding Modern Formats

Since I’ve been working on a tool in this space recently, I thought I’d write something up in case it helps folks unravel how to think about transcoding these days.

The tool I’ve been working on is EditReady, a transcoding app for the Mac. But why do you want to transcode in the first place?


After a day of shooting, there are a lot of people who need to see the footage from the day. Most of these folks aren’t equipped with editing suites or viewing stations – they want to view footage on their desktop or mobile device. That can be a problem if you’re shooting ProRes or similar.

Converting ProRes, DNxHD or MPEG2 footage with EditReady to H.264 is fast and easy. With bulk metadata editing and custom file naming, the management of all the files from the set becomes simpler and more trackable.

One common workflow would be to drop all the footage from a given shot into EditReady. Use the “set metadata for all” command to attach a consistent reel name to all of the clips. Do some quick spot-checks on the footage using the built in player to make sure it’s what you expect. Use the filename builder to tag all the footage with the reel name and the file creation date. Then, select the H.264 preset and hit convert. Now anyone who needs the footage can easily take the proxies with them on the go, without needing special codecs or players, and regardless of whether they’re working on a PC, a Mac, or even a mobile device.

If your production is being shot in the Log space, you can use the LUT feature in EditReady to give your viewers a more traditional “video levels” daily. Just load a basic Log to Video Levels LUT for the batch, and your converted files will more closely resemble graded footage.

Mezzanine Formats

Even though many modern post production tools can work natively with H.264 from a GoPro or iPhone, there are a variety of downsides to that type of workflow. First and foremost is performance. When you’re working with H.264 in an editor or color correction tool, your computer has to constantly work to decompress the H.264 footage. Those are CPU cycles that aren’t being spent generating effects, responding to user interface clicks, or drawing your previews. Even apps that endeavor to support H.264 natively often get bogged down, or have trouble with all of the “flavors” of H.264 that are in use. For example, mixing and matching H.264 from a GoPro with H.264 from a mobile phone often leads to hiccups or instability.

By using EditReady to batch transcode all of your footage to a format like ProRes or DNxHD, you get great performance throughout your post production pipeline, and more importantly, you get consistent performance. Since you’ll generally be exporting these formats from other parts of your pipeline as well – getting ProRes effects shots for example – you don’t have to worry about mix-and-match problems cropping up late in the production process either.

Just like with dailies, the ability to apply bulk or custom metadata to your footage during your initial ingest also makes management easier for the rest of your production. It also makes your final output faster – transcoding from H.264 to another format is generally slower than transcoding from a mezzanine format. Nothing takes the fun out of finishing a project like watching an “exporting” bar endlessly creep along.


The video industry has gone through a lot of digital formats over the last 20 years. As Mac OS X has been upgraded over the years, it’s gotten harder to play some of those old formats. There’s a lot of irreplaceable footage stored in formats like Sorensen Video, Apple Intermediate Codec, or Apple Animation. It’s important that this footage be moved to a modern format like ProRes or H.264 before it becomes totally unplayable by modern computers. Because EditReady contains a robust, flexible backend with legacy support, you can bring this footage in, select a modern format, and click convert. Back when I started this blog, we were mostly talking about DV and HDV, with a bit of Apple Intermediate Codec mixed in. If you’ve still got footage like that around, it’s time to bring it forward!


Finally, the powerful H.264 transcoding pipeline in EditReady means you generate beautiful deliverable H.264 more rapidly than ever. Just drop in your final, edited ProRes, DNxHD, or even uncompressed footage and generate a high quality H.264 for delivery. It’s never been this easy!

See for yourself

We released a free trial of EditReady so you can give it a shot yourself. Or drop me a line if you have questions.

Getting Old in a Maturing Industry

I recently returned from the 2013 NAB Expo in Las Vegas. For those who are unfamiliar, the NAB Expo is the biggest trade show and convention for the domestic film and broadcast industry. For one week every year, the Las Vegas Convention Center plays host to 100,000 film, video, and radio professionals, who come to check out the latest gear, learn what’s hot, and just generally hang out with a group of like-minded folks. It’s a “who’s who” of the business, and it’s far more massive than you can possibly imagine.

This was, as best as I can recall, my 12th year attending NAB. Some of those years have been quick hit-and-run visits (fly in in the morning, and out in the evening) and some have been incredibly intense week long affairs, building and manning the booth for Divergent Media. This year was somewhere in the middle – a week in Las Vegas, but only smaller events for the company, and plenty of time to explore the show.

In the aftermath of the show, I’ve been reflecting on the changes I’ve seen, and what it means about the industry.

When I first went to NAB in 2001, there was a sharp divide between the “indie” producer and the professionals. The desktop video revolution was still picking up steam, as DV cameras got better and “affordable” editing software gained pro features. Indie filmmaking involved a lot of clever repurposing, scraping by, and bending rules. Figuring out how to build your own steadicam approximation, repurposing Home Depot lights for your shoot, and assembling firewire drives from bare enclosures to save a few bucks.

There was precious little “prosumer” gear on the market – the pricing model still favored rental houses and big shops. Indies were largely ignored on the show floor. Apple and a few others saw where things were going, but the “big guys” were indifferent or downright disparaging.

Over the following four or five years, the “indie” part of “indie filmmaking” dropped off – everyone was an indie filmmaker, and everyone was a pro. Equipment became much more affordable and much better, software became cheaper, and everyone began to accept the new reality.

This was a particularly exciting time in the industry because we were right on the edge of what technology was capable of. Realtime effects, HD editing, multicam editing – you could see Moore’s Law in action. Each year, the quality of what could be done on a reasonable budget (or done at all) improved immensely.

The bursting of the financial bubble had a big impact on NAB, as it did on every trade show. Attendance dropped, and the pace of innovation slowed. But the bigger change is what began to happen around 2009, and built speed over the following years. The part of the industry we used to call “indie” grew up and became mature, and nothing replaced it to hurl rocks at the old guard. Today, innovation is no longer a matter of “last year that wasn’t possible and now it is,” or “last year this was $10k and now it’s $1k,” but rather “last year that took four steps and now it takes three” or “now it looks 5% better.”

Take, for example, the upstart camera manufacturer Red. Red was the star of the 2006 NAB show – they announced the Red One, a 4K “cinema” camera at a price that indies could potentially afford. They were aiming to disrupt the camera market, and filmmaking in general.

While Red had teething problems, by late 2007 it was a real product and accomplished a lot of what its creators set out to do. People who had been shooting films on $250,000 Sony F950s or actual film cameras began working with this $30,000 rig. Red bet on affordable pixel density, and for a while, they won.

The problem with disrupting a market is that if your competition survives the disruption, you have to compete on an ongoing basis. Now, in 2013, we have 4K (or higher) cameras from every manufacturer, at prices that in some cases substantially undercut Red. There are new kids on the block, like Blackmagic Design. And big vendors like Sony have caught up on pricing, marketing, and features. Whereas Red was once the star of the show, they’re not a bit of an also-ran.

The industry has matured in such a way that it gives filmmakers more choice for lower prices. You can’t pick a bad camera anymore. You can’t pick a bad editor. The industry has caught up with the market it serves. It’s no longer in need of a massive disruption. You don’t need clever tricks or dumpster diving – you just need talent.

It’s not just true in cameras of course – Final Cut Pro disrupted editing, Blackmagic’s revival of Resolve disrupted color correction, the LED revolution disrupted lighting, and obviously distribution is just one long stream of disruption.

The reality is that the film and video industry has grown up, and reached a point of stability. While there may be a new “indie filmmaking” revolution in the future, it doesn’t feel like it’s right around the corner. This is reflected in the NAB Expo. It feels like the demographics are skewing a bit older. I still feel like I’m one of the “young” folks there, even though I’m substantially older than when I first attended. There’s less razzle-dazzle – fewer vendors going to extremes to show up the vendor at the next booth. It’s less circus, less sexy, just business.

Divergent Opinions: NAB 2013

On this week’s episode of the media industry’s most influential podcast, Mike and I discuss our expectations for NAB 2013. We also dive into Apple’s new Final Cut Pro X marketing push and what it means for the NLE sector.

MOOCs: Solving the wrong problem

Over the last six months, the world of “e-learning” has been totally and completely overrun by the MOOC, or massive open online course. In my years involved with technology in higher education, I’ve never seen another concept suck the oxygen out of the academic technology space in this fashion. While not fundamentally new, the MOOC has totally dominated the e-learning conversation and dramatically shifted higher education agendas.

I’m a proponent of the core concepts of the MOOC. Reducing barriers to knowledge is a good thing. An education model based on keeping knowledge locked up in (forgive me) an ivory tower is fundamentally unsustainable in a digital world. I also believe that large, lecture-based classes can be inefficient uses of student and instructor time, and there are plenty of cases of subpar instruction in higher education.

The dramatic rise of the MOOC within the mainstream press and higher education discourse has resulted in traditional institutions rushing to join the party. It is this action that I believe is shortsighted, poorly reasoned, and potentially destructive.

Let me begin by articulating some of my assumptions. The term “MOOC” (we could have a whole separate article on how horrible the acronym is, but it’s too late now) means different things to different people. To some extent, it has become a “unicorn” term – a way for the uninformed to sound prescient without needing to back their words with knowledge. The types of courses I’m addressing here on are those taught in the traditional large lecture hall, usually introductory courses. These courses typically fill a lecture hall with disinterested freshmen, and often taught by a disinterested graduate student. Coursework generally consists of readings, discussion sections, and multiple choice and short answer tests.

The MOOC movement instead takes the lectures, breaks them into bite-sized chunks, and offers them presented by articulate, informed professors. Students would watch these lectures online, participate in online discussion with other students, and then take standard assessments. Because this type of course can be infinitely scaled, it can be opened to any and all interested parties – traditional students as well as those learning simply “for fun”. A single course can effectively support an unlimited number of students with negligible overhead.

I should note that, although I tend to think about these issues in terms of the institution I’m most familiar with – the University of Minnesota – I believe the lessons and missteps I’ll discuss are being repeated throughout the higher education system.

Dissecting the MOOC

As I stated at the beginning, I think the MOOC concept is perfectly fine. Taken on its own, the typical MOOC outlined above is a reasonable approach to replacing large lecture coursework. If it means lectures are better designed, more polished, and delivered with more passion and energy, that’s even better.

I do take issue with the notion that the MOOC is a righteous democratization of knowledge, opening it up to those learners simply interested for the joy of learning. I think this is a complete red herring argument, presented by MOOC backers to deflect valid criticism. The statistics bear this out: retention rates for MOOCs are consistently in the 10-15% range. It turns out that although most people think learning for fun sounds like a good idea, the reality is that it rarely happens. Moveover, the MOOC format does not lend itself to answering a specific question or providing a specific skill – training and tutorial organizations like and the Khan Academy are far better suited to these types of needs. If my end goal is to learn Photoshop or integral calculus, I’m unlikely to participate in an ongoing, sequenced course like a typical MOOC.

That said, I don’t otherwise find serious fault with the way MOOCs are being run by firms like Coursera and Udacity. If they’re able to sustainably produce a product that users are interested in, great.

The higher education response

The response from higher ed, on the other hand, is seriously flawed.

Let’s begin with the language. The companies behind the MOOC use language that assumes that traditional classroom instruction is broken. To hear them describe it, the typical lecture hall is a post-apocalyptic wasteland of bored students and depressed faculty, droning on, with little to no learning taking place. Rather than responding with an impassioned, evidence-based defense of this form of instruction, higher education has by and large accepted this characterization and decided that MOOCs are indeed the answer. Institutions have entered this discussion from a defensive, reactionary position.

Are large lecture courses perfect? Certainly not. In fact, I’m a huge proponent of the blended model, which seeks to shift lecture delivery to polished, segmented, online videos, with class time dedicated to deep dives and discussion. But I do think large lecture courses have value as well. A good lecturer can make the experience engaging and participatory, even in a lecture hall with hundreds of students. And, the simple act of trudging across campus to sit in a lecture hall for 50 minutes acts as a cognitive organizer – a waypoint on a student’s week. Keep in mind, these courses are typically made up primarily of eighteen year old freshmen, struggling with life on campus and figuring out how to organize their time and lives.

These large lecture courses almost always include a group discussion component. This is a time for students to participate in a mediated, directed discussion. More importantly, this is a time for the nurturing and growth that I believe is critical to the undergraduate experience. Seeing classmates, hearing their inane or insightful questions, listening to them articulate the things that they’re struggling with – these are all important but difficult to quantify parts of learning, which are not captured by a solitary or participation-optional online experience. Even a highly functional synchronous online discussion is inherently less likely to go down an unexpected rabbit hole in the exciting and invigorating way that a great classroom discussion can – even in a large lecture hall.

Instead of making the case for traditional courses and shifting the conversation to ways to make them even better (and more affordable), institutions have rushed to turn their own courses into MOOCs, and offer them to the world. This brings me to my next serious concern about this movement.

Why does anyone care about your MOOC? If the MOOC concept is taken to its logical end, the world needs a single instance of any given course. Institutions seem to be unwilling to acknowledge this. Instead, there’s an assumption they’ll build the courses and the world will beat a path to their door. Why would I choose to take a course from the University of Minnesota, when I can take the same course from Harvard or Stanford? Are institutions ready to compete at the degree of granularity this type of environment allows?

The rush to create MOOCs reminds me a bit of the little free library system. I lack the metrics to convincingly argue this point, but as an outside observer, it seems that people are generally more interested in creating the libraries than using them, and it feels like the number of libraries equals or exceeds the number of customers. I believe that the MOOC landrush risks resulting in a similar imbalance. If every school, from the prestigious to the second or third tier, rushes to offer MOOC forms of their courses, there is likely to be an abundance of supply, without the requisite demand.

Heating Buildings is Expensive. MOOCs are by and large the stereotypical “bubble” product. There’s little to no business model behind them, either for their corporate backers or participating institutions. Although that’s probably fine for the companies delivering the courses – their overhead costs are relatively low – it’s a huge issue for institutions with massive infrastructure and staff overhead. If we’re moving to a model where the cost of coursework is dramatically reduced, it presents existential threats for the other missions of the institution, or the very existence of the institution all together. While it’s assumed that institutions will continue to charge as part of the degree-granting process, nobody seems to think that a course delivered via a MOOC should be priced similar to a course delivered in a classroom.

How does the institution benefit? How does offering a MOOC make the institution a better place? Less expensive options for receiving course credit are certainly a benefit for students, but that is a separate issue. Higher education is far too expensive, but MOOCs are not the sole solution. In general, the value proposition for the University is esoteric – simply offering the course supposedly means the world will enhance the content by bringing a wider audience, and its existence will enhance your institution’s standing in the world. These are, at best, hopelessly idealistic. Because there’s no business model to speak of for MOOCs, Universities are left shouldering the costs of creating the courses with little to no expectation of having that value returned.

An alternative path

Having great courses delivered by great instructors with great instructional design is… great. I’d much rather take Introduction to Economics from Paul Krugman than a grad student. Having that type of content available is inarguably better for the world.

I believe the path forward is to leverage the best content, and combine it with the high-touch, in-person experience that makes undergraduate education so important, particularly for traditional students. Mediated discussions, in-person office hours, and writing assignments graded by people with (ostensibly) actual writing skills are the types of growth activities that create the high functioning, well-rounded people our society needs.

It’s also crucial for higher education to begin pushing back against the language of the broken classroom. Although institutions are indeed broken in innumerable ways, by and large, instruction actually works pretty well.

It’s critical as well that a clear distinction is drawn between MOOCs and online education in general. Along with Jude Higdon, I teach an online course which is in many ways the anti-MOOC. Our students do large amounts of writing, which is copiously annotated by the instructors. Students participate in synchronous chat sessions and office hours with the instructors. Although lecture content is reused, the interpersonal interaction is real, genuine, and frequent. This concept is, by design, not scaleable. But I believe the benefits in terms of breadth and depth offered to students by this experience are demonstrably better than that offered by a MOOC. Institutions need to be honest about these tradeoffs.

A MOOC is not a magic bullet. It will not solve higher education’s substantial woes. It will create new woes.

Your MOOC will almost certainly not make a dent in the universe. The world will not beat a path to your door, and you still need to pay to maintain the doors.