Mesos and YARN are two powerful cluster managers that can play host to a variety of distributed programming frameworks (Hadoop Map-Reduce, Dryad, Spark, and Storm) as well as multiple instance of the same framework (e.g., different versions of Hadoop). Both are concerned about optimizing utilization of cluster resources especially in terms of data locality of data distributed around the cluster. Google's paper on Omega, their own cluster scheduling system, dubs Mesos a two-level scheduler, which provide some flexibility by having a single resource manager offer resources to multiple parallel, independent schedulers. YARN is considered a monolithic scheduler since independent Application Masters are only responsible for job management and not scheduling. Scheduling is the essence of efficient Big Data processing. However, where do these two systems differ?
Tuesday, December 9, 2014
Friday, December 5, 2014
Container Virtualization Options
 | |
| photo by sioda | via PhotoRee |
Looks like the container virtualization space is becoming a little more interesting this week. Previously, Docker was the only more or complete standard container implementation (with definition of image, image creation, and container start/stop management). There was Canonical's LXD, it didn't seem to be garnering nearly as much attention and support since it was only announced a month ago. However, with the Docker and CoreOS organizations starting to encroach on each other's territory, the CoreOS community has released an early version of their own container runtime, Rocket. On the balance, Docker has moved into the container cluster orchestration and management space with Docker Swarm and Docker Compose, the latter being still in the design stage.
