Parallel Data Types

True parallelism can be achieved via a wide variety of avenues. Instruction-level parallelism arguably offered a huge value at one point when processor architectures bulked up to execute machine instructions in parallel whenever possible including when instructions can be reordered or delay slots can be filled. All this could have been done without an application programmer's involvement or knowledge, hence it promised the benefits of parallelism for most programs out-of-the-box without modifications. The onus, instead, was on the processor architects and compiler developers to take advantage of such parallelism, which sometimes turned out to be a tall order. Task parallelism, in contrast, typically demands a lot more from application programmers since exposing and taking advantage of parallelism becomes very domain-specific. The focus of parallelism for task parallelism depends on balancing the loads of execution threads. Runtime support can help fill in the gaps here by applying advanced scheduling techniques such as work stealing, but ultimately what constitutes a thread (i.e., how to partition the program) and how to mediate communication and sharing between threads is often up to the programmer. Another way to achieve parallelism at a fine-grain level is to implement and expose a parallel data type in a language, runtime, or distributed computing framework. Apache Spark achieves this using its RDD (Resilient Distributed Dataset) abstraction which can copy data from Scala Seq, Java Collection, and Python iterables. Hadoop uses Java Iterable for MapReduce. For modern distributed computing environments, the pipeline itself might be any DAG (more general than map-reduce), but ultimately parallelism stems from the data representation, hence data parallelism. Note that of all the forms of parallelism it is data parallelism that ultimately has demonstrated straightforward and efficient scaling to truly large problems.

Notes from the MacQueenFest

I had the excellent opportunity to attend the MacQueenFest a couple of weekends ago in honor of David MacQueen. The venue provided interesting insight into what the alumni of the ML community was up to these days. Many of the slides are now up on the website. The talks were scheduled roughly chronologically based on Dave's contributions. Most were looking forward as much as they were considering the historical significance of the contributions.

Memory Management and Concurrency in C++

Despite the flurry of activity from the Boost and C++11 standards committee, two crucial things remain quite challenging: efficient memory management and concurrency. What started me down this train of thought was some good old code reading. I encountered plain old vectors of pointers. There are apparently a handful of alternatives to this. Boost implements ptr_vector that keeps track of the lifetimes of all its elements as a unit. vector<shared_ptr> is another possibility. The core of the problem is that since vectors' underlying representation is a pointer, and one cannot get a pointer to a reference (the assumption being one wanted the efficiency of reference-passing but with the convenience of STL containers).

More after the jump

Programming Language Gotchas

Looks like I spoke too soon. The October rally was looking a little long in tooth, and a little warning from the credit agencies was enough to get the market to go risk off at least for this afternoon. At least yields did not decline too much (2.01% for the 10-year according to MarketWatch). The Dollar Index (DX), however, has certainly been in an uptrend since the end of October trough.

Coming from the Standard ML world of languages makes me take many things for granted. It also gives me a probably pretty narrow perspective. Hacking on my trading analysis tools in other languages helps me broaden that somewhat, though I have certainly worked in the mainstream languages long before I encountered ML and its brethren. I would like to keep a record of these gotchas as I encounter them. I do not intend this as value judgments but rather a documentation of noteworthy differences from a higher-order typed programming language perspective.