Tuesday, December 9, 2014

Apache Mesos and Hadoop YARN Scheduling

Mesos and YARN are two powerful cluster managers that can play host to a variety of distributed programming frameworks (Hadoop Map-Reduce, Dryad, Spark, and Storm) as well as multiple instance of the same framework (e.g., different versions of Hadoop). Both are concerned about optimizing utilization of cluster resources especially in terms of data locality of data distributed around the cluster. Google's paper on Omega, their own cluster scheduling system, dubs Mesos a two-level scheduler, which provide some flexibility by having a single resource manager offer resources to multiple parallel, independent schedulers. YARN is considered a monolithic scheduler since independent Application Masters are only responsible for job management and not scheduling. Scheduling is the essence of efficient Big Data processing. However, where do these two systems differ?

Because Mesos and YARN have to host a wide range of frameworks with very different workload characteristics and resource requirements, they delegate actual scheduling of tasks under a single framework to that framework. In contrast, Kubernetes and other container cluster managers have a single scheduler or set of schedulers to choose from for all workloads. Although the general schedulers implemented in frameworks will likely be similar (min-max fairness and priority scheduling), Mesos and YARN take different approaches to partitioning resources. YARN clients need to submit Resource Requests upfront to the YARN ResourceManager (essentially the master in this architecture). In contrast, Mesos's primary mechanism is to have the Mesos's master present resource offers to each hosted framework which is then free to reject the offer any number of times. Google's Omega paper calls this form of scheduler management pessimistic concurrency. As an optimization, a hosted framework can present a filter to the master to pre-filter any potential offers that would have definitely been rejected by the hosted framework. YARN's Resource Manager itself handles node failures whereas Mesos delegates such failures to each framework's own policy. Because Mesos defers somewhat more of the cluster management responsibilities to frameworks, it also requires somewhat less configuration than YARN.

In terms of similarities, there are many. Both cluster managers implement some form of master-slave architecture with hot backup masters in case of master failure. Mesos does so using Zookeeper to provide for leader election whereas YARN implements its own. In terms of node fault tolerance, both will restart nodes that have not reported back after a configurable grace period. Both have some support for Docker containers. With YARN, Docker containers can only serve as executors whereas in Mesos they might either be an executor or a task (meaning one can submit a container to a master to run on slaves).

Rough Mapping of YARN and Mesos Terminology

YARNResource ManagerApplication MasterNode ManagerContainer*
MesosMasterFramework SchedulerSlaveFramework Executor

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.