Tuesday, November 24, 2015

HPC Convergence and the Holy Grail Rapid Prototyping

In the field of High Performance Computing (HPC), there is a constant and obscured debate between the old traditional toolkit users of Fortran, Cobol, Java and .NET, representing the collective corporate / enterprise and academic / government laboratory sectors of the HPC world, and the modern rapid prototypers using web stack environments of Rails, Clojure, Go, Django, and the myriad Javascript web frameworks. In practice, however, the true debates aren’t aired, because they exist in completely separate worlds: the dotcom 2.0/3.0 new generation of the west coast and the traditional rapid prototypers of the IT/defense/enterprise world.

The new generation equips themselves with modern tools, including Ruby on Rails, Django, and jQuery without pause. Meanwhile, the traditional rapid prototypers equip themselves with the tools of .NET, Fortran, Java, and other “interpreted languages”; to this community, anything that isn’t C or assembly is a shortcut (C++ is for the lazy).

Enter the brave new world.

In the spirit of the old DARPA research conducted as part of the “High Productivity Computing” initiative, I propose a new framework for thinking about rapid prototyping. Instead of focusing on the languages themselves, I’ll focus on the cross-talk and intersections between languages. The result is what I’ll call the “holy grail of pragmatic computing” (aka rapid prototyping) – a development environment in which the fastest possible mockup of a concept can be delivered in a language-agnostic environment. This is only feasible if the proper collection of languages and API’s are known.

I argue that the language chosen for rapid prototyping matters far less than whether that language can tap into the multiplicity of optimized toolkits written in other languages. No one in their right mind writes hacking exploits in Clojure; no one writes high level number theory proofs in C; and no one writes graphic intensive games in Excel VBA.

I therefore hold the following truths to be self-evident by practitioners:

  • Excel is best environment for manipulating datasets less than a gigabyte large (aka most around today)
  • Clojure is the best environment for developing high level data transmutation and system block level algorithms
  • Python is terrible but has so many libraries and modules written for it, it has become unavoidably implicated in any prototype
  • Mathematica is the superior applied mathematic symbolic toolkit providing most immediate delivery of high end computational functions

The “truths” above will offend most programmers. If, however, you suspend disbelief and agree, the following matrix identifies the most useful tools around to enable this rapid protoyping:



Equipped with this simple matrix of cross-talk enabling APIs and tools to permit programs written in one language to utilize modules and libraries written in other languages, the modern programmer can achieve levels of efficiency optimization unthinkable to the former generation of developers.