tag:blogger.com,1999:blog-27400671762257822152024-02-21T09:23:51.715-08:00Programming languages, Cloud, and Financial MarketsInsights into financial markets and technologyGeorgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.comBlogger145125tag:blogger.com,1999:blog-2740067176225782215.post-82114099383775030732023-06-08T07:00:00.001-07:002023-06-08T07:00:00.161-07:00Bug Lifecycle<p>
Back in the International Conference on Software Engineering (ICSE) 2010, Philip Guo et al published a study on bug resolution in a paper "Characterizing and Predicting Which Bugs Get Fixed: An Empirical Study of Microsoft Windows". The paper investigates the factors that influence bug-fixing decisions in the development of Microsoft Windows, particularly Windows Vista and Windows 7. The authors aimed to understand why some bugs in the Windows operating system are addressed promptly while others remain unresolved for an extended period. They conducted an extensive empirical study by analyzing a large dataset of bug reports and their associated properties from the Windows bug tracking system.</p>
<p>
The paper provides several key findings based on their analysis:
<ol>
<li>Bug Characteristics: The study found that certain bug characteristics influence the likelihood of them being fixed. Bugs that are easier to reproduce, have clear descriptions, and affect a wider user base are more likely to be addressed promptly.</li>
<li>Bug Severity: The severity of a bug plays a significant role in determining the likelihood of it being fixed quickly. High-severity bugs, which have a substantial impact on user experience or system stability, are given higher priority and are more likely to be fixed sooner.</li>
<li>Developer Expertise: The experience and familiarity of developers with specific parts of the codebase influence bug-fixing decisions. Developers tend to fix bugs related to areas they are more knowledgeable about, resulting in variations in bug-fixing rates across different components of the system.</li>
<li>Bug Reporting Quality: The quality of bug reports, including the level of detail, reproducibility, and clarity of description, affects the likelihood of a bug being fixed. Reports that provide more precise information are more likely to receive attention and prompt resolutions.</li>
<li>Bug Activity and Age: Bugs that receive more comments, indicating active discussions and user interest, tend to be fixed faster. Additionally, the age of a bug influences its chances of being fixed, with older bugs often being prioritized to reduce the backlog.</li>
</p>
<p>
Based on these findings, the authors developed a prediction model to estimate the probability of bug fixes. They used machine learning techniques to train the model, incorporating various bug characteristics and contextual factors. The model showed promise in predicting which bugs are more likely to be fixed promptly.
The paper presented a model based on some definition of bug reporter reputation, frequency of bug edits, bug reassignments, geographical distance, and reopenings. They present both the predictive and descriptive statistical models
</p>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-15073294232905531242023-06-07T04:15:00.010-07:002023-06-07T04:20:26.137-07:00How to do a code review, Part 2<p>Microsoft several years ago published <a href="https://queue.acm.org/detail.cfm?id=3292420">a conversational article on their case study on code review</a> as well as their own in-house code review tool CodeFlow.</p>
<ol>
<li>Code Review Challenges: The article highlights several challenges associated with code reviews, including time constraints, difficulty in coordinating reviewers, lack of context, and limitations of traditional review methods like in-person meetings or email-based reviews.</li>
<li>Asynchronous Code Reviews: CodeFlow addresses the challenges by providing a platform for asynchronous code reviews. This allows developers to review and provide feedback on code changes at their own convenience, regardless of time zones or physical location.</li>
<li>Distributed Collaboration: CodeFlow enables distributed collaboration by providing a centralized platform where developers can review, comment, and track the progress of code reviews. It eliminates the need for manual coordination and facilitates efficient communication between reviewers and authors.</li>
<li>Code Quality Improvement: Code reviews play a crucial role in improving code quality. By involving multiple reviewers, CodeFlow helps identify defects, bugs, and design flaws early in the development process. This leads to better overall code quality and reduces the likelihood of introducing errors into the codebase.</li>
<li>Knowledge Sharing: CodeFlow promotes knowledge sharing among developers. Reviewers gain insights into different codebases and programming practices, fostering a culture of learning and continuous improvement. It also helps distribute knowledge across teams and reduces reliance on individual developers.</li>
<li>Iterative Review Process: CodeFlow supports an iterative review process where developers can address feedback, make changes, and have subsequent rounds of review. This promotes collaboration, allows for further refinement of code changes, and encourages constructive discussions between authors and reviewers.</li>
<li>Metrics and Insights: The article presents statistical data on CodeFlow's usage, including the number of code reviews conducted, average review duration, and the number of code changes accepted or rejected. These metrics provide insights into the effectiveness and efficiency of the code review process.</li>
</ol>
Check out <a href="http://www.typedynamic.com/2017/07/how-to-do-code-review-part-1.html">part 1 of these series on code review</a>.Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-42625095957738174892017-07-10T11:00:00.000-07:002017-07-10T11:00:15.783-07:00Corporate Governance Today<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpdHAag1TV4ZtJpIPlhZPLAFz-76PplI8b7LoyJFp-kFFwwzO_wqEJanqdaYpWnLMAHObvekKeJjpSURJR3-dUs8kQ9Ynqge1Gy4g7gvca30yNYYCieK6SySl2zK7iDXcMQIPIT6eAxhJk/s1600/pexels-photo-2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpdHAag1TV4ZtJpIPlhZPLAFz-76PplI8b7LoyJFp-kFFwwzO_wqEJanqdaYpWnLMAHObvekKeJjpSURJR3-dUs8kQ9Ynqge1Gy4g7gvca30yNYYCieK6SySl2zK7iDXcMQIPIT6eAxhJk/s320/pexels-photo-2.jpg" width="320" height="213" data-original-width="1600" data-original-height="1067" /></a></div>
<p>
Once upon a time, companies were managed by founder-owners, engineers, gritty tradesmen, and the like. The current norm of companies managed by a class of professional managers is a relatively recent development. Once upon a time, the dream of an upstart company, especially a tech company, was to go public and become a publicly traded company. A couple of market crashes later and a rather unique interest rate environment has turned this aspiration upside down and many companies have chosen to delay initial public offerings. As of late, the pace in IPOs has picked up somewhat, but it still pales in comparison to the heyday of tech IPOs. One performance all publicly traded companies must have is the shareholders' annual meeting. There is quite a contrast from company to company in how they carry this out. For Berkshire Hathaway, it is a celebratory confab of the giants of industry and finance coming before the oracle of Omaha Warren Buffet. For most companies, it is a formal requirement that they rather do without. In some cases, annual meetings have become outright adversarial. Having already to file numerous SEC-required filings and give earnings calls for the benefit of Wall Street analysts, yet another venue where shareholders, the purported fractional owners of companies, and influencers can raise questions is probably few management teams' idea of fun. It seems to be all too easy for an annual meeting to become adversarial, whether it be due to an impending say-on-pay vote, some kind of environmental or social concern, or just plain input from the shareholders on how a company can tack to become more efficient, profitable, and fast growing. Few teams want to hear from the shareholders because the best input from shareholders always seems to be no input from the perspective of the management. With a large part of the public shares of companies held by mutual funds and ETFs, which in turn are held in 401k accounts, this affects many of us. Nearly all the time, funds carry out a perfunctory according to management recommendation without any consideration or else abstain from voting. This works very much in the favor of management since they can count on funds to support share value, their share value, while not having to worry about large shareholders attempting to get in their say. When did corporate governance become so adversarial and arguably dysfunctional?
</p>
<p>
This situation also affects tech companies. Though some of the leading tech companies are nominally run by engineers, most are backed by a senior leadership team of professional managers supported by legions of more professional managers. It's not to say that management's job is easy. It is very challenging and sometimes you are put in a no-win situation. However, the fact that communication has broken down or started to fray at the corners between management and their nominal bosses, the shareholders, points to some kind of sea change. Even the entry of activist investors such as Paul Singer's Elliot Management, Carl Icahn, and Third Point are a relatively new phenomenon.
</p>
<p>
I have written previously about <a href="/2011/12/shareholder-meetings.html">shareholder meetings</a> and how some small investors such as Matthew Rafat are trying to get their say in.
</p>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-86337629013350801552017-07-09T09:00:00.001-07:002017-07-09T09:00:00.521-07:00Top Posts<ol>
<li><a href="/2011/11/share-buy-backs-part-1.html">Share Buybacks</a> is a four-part series that explores the historical performance of the share buyback strategy which seeks to overweigh companies with large share buyback programs.</li>
<li><a href="/2011/12/buybacks-and-dividend-growth.html#more">Buybacks and Dividend Growth</a> explores the debate between proponents of each strategy.</li>
<li>The <a href="/2012/02/saving-up-for-college-tuition-and.html">Saving up for college tuition</a> series muses on the value and underlying frictions found in college savings and investments accounts.</li>
<li><a href="/2012/01/stock-price-limits-and-adrs.html">Stock Price Limits and ADRs</a> surveys the research literature on the impact of stock price limits on price discovery.</li>
<li><a href="/2013/02/max-drawdown.html">Max Drawdown</a> recaps a study on the performance of equities and the best and worse case scenarios for buy-and-hold investors during the height of the financial crisis.</li>
<li><a href="/2012/01/market-indexing.html#more">Market Indexing</a> explores fundamental index including a brief history and the pros and cons.</li>
<li><a href="/2012/04/probabilistic-counter.html">Probabilistic counter</a> goes over some of the seminal papers describing the classic efficient non-comparison-based algorithms.</li>
<li><a href="/2012/01/memory-management-and-concurrency-in-c.html">Memory Management and Concurrency in C</a> surveys the costs and benefits of various memory management techniques in software.</li>
<li>Programming language theory meets popular programming languages in <a href="/2012/06/formal-language-semantics.html">Formal Semantics for the Top 5 Programming Languages</a>. Over the last couple of decades, the research community has built an amazing amount of technical machinery in the form of ever fancier type systems and logics to formalize the semantics of programming languages. Most of the time, the subject under study is some niche language that is relatively constrained to admit formalization. However, in the above mentioned post, researchers tackle the fickle and messy popular programming languages head on to try to bring reason and order to a disorderly world.</li>
<li><a href="/2012/06/scientific-method-and-epistemology.html">Scientific method and epistemology</a> presents a 10,000 foot level view of the philosophy of science and knowledge.</li>
</ol> Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-61433456579086851132017-07-09T09:00:00.000-07:002017-07-09T09:00:43.797-07:00How to do a code review, Part 1<p>
Code review is a practical and important skill that truly distinguishes the highly experienced engineer. Knowing what are the common pitfalls, where to look for them, and how to understand someone else's code quickly is an invaluable skill to have. It is also a skill that is rarely taught in school. To really ramp up your programming skills, get out there and read other people's code. Reading high quality code is a good learning experience. Occasionally reading poorly written code is also insightful. What are the most common errors and where do you find them? Through code reviews, software engineers can spread knowledge through their teams as well as improve consistency and code quality. For the interviewee, you have to review your own code, so it helps to know what are the things you need to double check. One great way to evaluate a team's culture is to probe and understand their code review practices or lack thereof. Below, I will cover some of the best practices and provide a few pointers on where you can get started.
</p>
<a href="http://www.typedynamic.com/2017/07/how-to-do-code-review-part-1.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-27616136820103293602017-07-02T20:18:00.001-07:002017-07-02T20:18:31.018-07:00Platform-as-a-Service Providers<p>
A overwhelming amount of attention and resources has been focused on Infrastructure-as-a-Service (IaaS) clouds. IaaS clouds are in a sense more flexible but requires considerably more setup and configuration than Platform-as-a-Service (PaaS) clouds. IaaS clouds can serve both for webapp hosting as well as Hadoop-style cluster computations. PaaS clouds are clearly targeted at webapp hosting. PaaS cloud hosts usually provide a wide variety of supporting services (e.g., payment gateways, monitoring, CRM) to help developers focus on core business logic, but often businesses have found out that PaaS is too restrictive, requiring one to give up too much control over the webapp architecture. In particular, PaaS often requires one to select from a limited set of data stores and load balancers. Sometimes, PaaS requires webapps to be written against a particular proprietary API, potentially a restricted and even proprietary data store. This would make something rather simple such as a deploying a Wordpress site a non-starter.
</p>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-69460753967183829952017-07-01T13:14:00.001-07:002017-07-08T11:35:19.495-07:00Resources<p>
<ul>
<li>
<a href="https://www.whatsopening.com">What's Opening in your neighborhood</a> scours the web for terrific restaurants that are coming soon to your neighborhood.
</li>
</ul>
</p>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-86001239585451334842016-06-23T06:33:00.004-07:002017-07-04T14:50:09.626-07:00Explaining Market Crashes<p>
The body of academic theories for market crashes explores a number of possible explanations. Why do markets crash? There may be some event that sets everything in motion, but why and when does a decline turn from an ordinary slide to a crash? Didier Sornette wrote a book and did a <a href="http://www.ted.com/talks/didier_sornette_how_we_can_predict_the_next_financial_crisis">TED talk</a> on the subject. Scholars as illustrious as Fischer Black and Myron Scholes weighed in on the subject. There are four main theories addressing this phenomenon:
<ol>
<li>Leverage effects: Drop in prices increases leverage both operating and financial thus exacerbating volatility when businesses and investors have to raise capital to cover leverage or by reducing leverage</li>
<li>Volatility feedback: When bad news arrives, the risk premia magnifies the direct effect of the news</li>
<li>Stochastic bubbles: Crash occurs when a buble pops thus resulting in a low-probability event that produces large negative returns</li>
<li>Investor heterogeneity: Different investors have varying constraints when it comes to short-sales. The more bearish group of investors subject to short-sales constraints may just sell all their sales, a suboptimal solution, and thus their information is not fully included in the market</li>
</ol>
The oldest theory is focused on leverage effects. The idea is that a drop in prices raises operating and financial leverage thus exacerbating volatility. This theory is articulated by Fischer Black and Myron Scholes in 1973 "The Pricing of Options and Corporate Liabilities" in the Journal of Political Economy.
Andrew A. Christie further explores the idea in "The Stochastic Behavior of Common Stock Variances--Value, Leverage and Interest Rate Effects" in the Journal of Financial Economics 1982.
</p>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-21079959656531977542016-04-06T17:02:00.000-07:002016-04-06T22:52:26.525-07:00Cloud Providers in China<p>
The cloud business is slowly but surely picking up in China as it has in the West. Out of the major multinational providers, only Microsoft Azure offers comprehensive service in China. Amazon in 2014 has introduced a closed limited program for region in Beijing, but this service excludes common services such as Elastic Beanstalk. China, however, has its own homegrown cloud services, principally SinaCloud and AliYun from Sina and Alibaba respectively. Baidu offers a PaaS service but not a full fledged IaaS. Unlike US cloud services, most Chinese cloud services are pay by the month instead of pay by the hour. This arrangement is quite limiting since one would be unable to scale down costs below the monthly unit. This truly adds up when one is talking about thousands of instances.
</p>
<a href="http://www.typedynamic.com/2015/10/cloud-providers-in-china.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-91224340325353894232016-02-02T09:21:00.003-08:002017-07-08T12:24:15.277-07:00Investigating dislocations in the oil complex<p>
The oil and basic material complex has suffered quite a downturn in 2015 and also in the beginning of 2016. Equity prices have declined dramatically. Oil and basic material company bonds have also been under quite a lot of pressure from downgrades from the credit rating agencies. Even large-cap companies have seen the yields they have to pay skyrocket. As the truism goes, when market volatility goes up, correlations all head to 1, since everyone sells everything. Having noticed HAL, BHP, and COP bonds going for 4.5% to 6%+, which is a huge spread to Treasuries, one wonders how severe of a downturn the market is pricing in.
</p>
The following is the correlation of the changes in bond yield (ought to be inversely related to bond price) and USO returns over the period January 4 to February 1.
<table>
<thead><th></th><th>Bond Yield</th><th>Equity Price</th></thead>
<tr><td>HAL</td><td>-0.064</td><td>0.68</td></tr>
<tr><td>COP</td><td>-0.62</td><td>0.79</td></tr>
<tr><td>BHP</td><td>-0.78</td><td>0.67</td></tr>
</table>
<p>
Unsurprisingly, each of these companies equity prices are highly correlated to oil prices at this juncture. Even BHP, which is mildly removed from the oil complex, since it works in many basic materials and mining beyond oil, is highly correlated. It turns out that the bond yields of these otherwise investment grade bonds are also moving with the oil prices except for the case of HAL.
</p>
Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-60775612796118080852014-12-29T06:00:00.000-08:002014-12-29T06:00:03.845-08:00Understanding Cloud Host Pricing, Part 2<p>
In the past few years, there has been a movement to standardize cloud compute resource measurements in order to make way for public trading of compute resources. The idea is simple, but execution may be complicated: each company can run something like OpenStack and rent off underutilized compute resources and these resources can be further trading on public exchanges to enable companies to hedge for price spikes. Along these lines, Amazon was quite early in introducing the <a href="http://aws.amazon.com/ec2/purchasing-options/reserved-instances/marketplace/">Reserved Instance Marketplace</a>. A public trading of standardized compute units will enable smaller organizations to monetize underutilized assets. This model is not without its challenges. Compute resources have many aspects that distinguish them. Performance may vary dramatically. In this post, I investigate some of the smaller cloud hosts and their prices.
</p>
<table>
<tr><th>Provider</th><th>Minimum Unit ($/hr)</th><th>Memory (GB)</th><th>Instance Storage (GB)</th><th>Persistent Block Storage ($/GB/mo)</th></tr>
<tr><td>HP Helion</td><td>0.03</td><td>1</td><td>10</td><td>0.10</td></tr>
<tr><td>IBM SoftLayer</td><td>0.04</td><td>1</td><td>25</td><td>0.10</td></tr>
<tr><td>Oracle Cloud</td><td></td><td>1.8</td><td></td></tr>
<tr><td>CloudSigma</td><td>0.0319</td><td>1</td><td>1</td><td>0.14 SSD</td></tr>
<tr><td>GoGrid</td><td>0.02-0.03</td><td>0.5</td><td>25</td><td>-</td></tr>
<tr><td>DreamHost DreamCompute</td><td>0.0264</td><td>2</td><td>25</td><td>-</td></tr>
<tr><td>Internap</td><td>0.04</td><td>1</td><td>20 SSD</td><td>0.30</td></tr>
</table>
Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com2tag:blogger.com,1999:blog-2740067176225782215.post-62875524715334326072014-12-27T06:00:00.000-08:002015-01-06T22:56:15.698-08:00Top 5 Gotchas When Running Docker on a Mac<p>
Running Docker on Mac is meant to be a convenience but the fact that Docker on Mac is a 2nd class citizen shows up every now and then. Since Docker is based on Linux cgroups, it cannot and does not run natively on MacOS X. Instead, Docker runs on Macs by using boot2docker, a shim that boots up a whole VirtualBox VM on which one will actually run Docker. Running Docker inside of a VM on Macs complicates things quite a bit.
</p>
<a href="http://www.typedynamic.com/2014/12/top-5-gotchas-when-running-docker-on-mac.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-55852341373195519132014-12-26T06:00:00.000-08:002014-12-26T06:00:07.733-08:00Understanding Cloud Host Pricing, Part 1<iframe src="https://www.flickr.com/photos/maddog78/2212457767/in/photolist-6EkSKG-9zbjDz-5p9nh3-4PVqwm-6EgGEF-9RqBfq-4nXfwv-9RQkdN-dmjSdA-84o2ER-ehyhPC-7oFYrc-4nvqBz-csMQXb-nRegFf-6Xw3vL-ntS23C-oCs3tU-9hYBcr-6Xw3Mq-b6wkkP-7u8sVQ-ozKQMu-yqcg-6fTmk7-d3W8PG-5MoveP-jtMME6-XEMwS-d2hjzJ-cCizvm-oqJqTF-77YfPx-8gszbW-6MUZEZ-dhtwtk-p2ZZVZ-7oFU8a-3SvhB-e5wER1-bzm3vU-6EkzCQ-nXRyaB-2SeRXT-7oFYoX-84r6Mo-iDjqK-6DDidk-mWQaAC-oMWsyY/player/" width="500" height="500" frameborder="0" allowfullscreen webkitallowfullscreen mozallowfullscreen oallowfullscreen msallowfullscreen></iframe>
<p>
The pricing schemes of the top cloud infrastructure-as-a-service (IaaS) providers are rather complicated. They cannot be compared directly since their performance characteristics vary. Moreover, the differences in costs of instances, storage, and bandwidth may offset each other. For example, one provider A's storage costs may be greater than provider B but B's instance costs may be greater. Some providers bill for instances by the minute (Azure and Google Computer Engine) whereas others bill by hour (rounded up), such as AWS. Some providers charge for storage by the TB (Azure) whereas others charge by GB/month or sometimes GB/hour (Rackspace). Consequently, what constitutes to the best deal in terms of cloud hosting depends on your specific workloads and storage needs. In this series, we will investigate the various aspects of cloud host pricing from the major providers: Amazon AWS, Microsoft Azure, Google Compute Engine, DigitalOcean, and Rackspace.
</p>
<a href="http://www.typedynamic.com/2014/12/understanding-cloud-host-pricing-part-1.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-88641358234973409892014-12-17T07:41:00.000-08:002017-08-02T19:46:04.979-07:00Retry Pattern<p>
A common design pattern in fault-tolerant distributed systems is the retry pattern. A given operation may experience a variety of failures:
<ol>
<li><h2>rare transient failures</h2> (e.g., corrupted packet) can be recovered from immediately and thus should retry immediately </li>
<li><h2>common transient failures</h2> (e.g., network busy) can retry after waiting for a period of time (possibly with exponential backoff</li>
<li><h2>permanent failure</h2> should not retry, bail out and clean up</li>
</ol>
Of course, the final case is that the operation succeeds and the function must do some work to address that. This is an interesting design pattern not only for distributed systems but also fault-tolerant systems in general. For example, high-performance JavaScript engines have parsers and tokenizers that must be robust to various failures. In fact, it is one example where large systems have used multiple exit points, more complicated control flow for which C-based programs may use gotos.
</p>
<a href="http://www.typedynamic.com/2014/12/retry-pattern.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-11945563144869943402014-12-10T05:04:00.003-08:002014-12-10T05:04:59.044-08:00Docker Image for SML/NJ<p>
The various Linux distros package repos carry very outdated versions of the SML/NJ compiler. This <a href="https://registry.hub.docker.com/u/commonlisp/smlnj/">Docker image</a> builds the latest official SML/NJ release.
</p>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-29580798067839578982014-12-09T06:00:00.000-08:002014-12-18T08:18:50.826-08:00Apache Mesos and Hadoop YARN Scheduling<p>
Mesos and YARN are two powerful cluster managers that can play host to a variety of distributed programming frameworks (Hadoop Map-Reduce, Dryad, Spark, and Storm) as well as multiple instance of the same framework (e.g., different versions of Hadoop). Both are concerned about optimizing utilization of cluster resources especially in terms of data locality of data distributed around the cluster. Google's paper on Omega, their own cluster scheduling system, dubs Mesos a two-level scheduler, which provide some flexibility by having a single resource manager offer resources to multiple parallel, independent schedulers. YARN is considered a monolithic scheduler since independent Application Masters are only responsible for job management and not scheduling. Scheduling is the essence of efficient Big Data processing. However, where do these two systems differ?
</p>
<a href="http://www.typedynamic.com/2014/12/apache-mesos-and-hadoop-yarn-scheduling.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-5166616858487587082014-12-08T06:00:00.000-08:002014-12-09T05:53:14.916-08:00Alternatives to Docker: LXD and Rocket<p>
Two recently announced alternatives to the Docker Linux container runtime LXD and Rocket aim to offer some interesting value propositions. Before I get to the details, let's first identify what use cases and aspects of the Docker container runtime of interest here. Docker identifies a few major categories of use cases: continuous integration, continuous delivery, scaling distributed applications, and Platform-as-a-Service. The former two are DevOps use cases. At this point, Docker pretty much has a lock on DevOps use cases. Moreover, neither of the would-be competitors truly target DevOps. It becomes obvious when you consider that LXD was intended to run on OpenStack Server environments and Rocket on CoreOS/fleet (though it is not necessarily tied to CoreOS). Docker runs on workstation environments and even on top of VirtualBox via boot2docker to support DevOps functionality on MacOS X. The more competitive aspect is the cloud infrastructure one. Here, Docker is competing with a wider range of technologies to support scaling on the cloud and providing PaaS functionality. This is the market where LXD and Rocket would operate. This is also the area where hypervisors have reigned.
</p>
<a href="http://www.typedynamic.com/2014/12/alternatives-to-docker-lxd-and-rocket.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-60998426603873125132014-12-07T06:00:00.000-08:002017-07-04T14:51:35.324-07:00How to search for great programmers, Part 2<p>
Aline Lerner of Trialpay recently posted <a href="http://blog.alinelerner.com/resumes-suck-heres-the-data"> statistics on an experiment about resume review</a>. The conclusion was that recruiters, engineers, and just about everyone score resumes all over the place and therefore resumes have weak signal value. The claim was that the strongest signal in the resume was the number of typos. Although the study seems extensive, I think there are a number of weaknesses in the experimental design. One weakness was acknowledged: the ground truth is the author's own subjective evaluation of the candidates. Another weakness was how the survey questions were somewhat misleading in the first place. The questionnaire asks "would you interview this candidate" and yet this was compared with the ground truth of "will the candidate perform well on the job or technical interview". As I alluded to in an <a href="http://www.typedynamic.com/2014/11/how-to-search-for-great-programmers.html">earlier post</a> in this series, the role of a resume is to help filter for red flags and to guide the formal interview, not to determine whether a candidate is a star performer by itself. The fact of the matter is, a resume is a self-reported synopsis of a candidate's track record. To evaluate a candidate, I would think track records are important, as is potential.
</p>
<a href="http://www.typedynamic.com/2014/12/how-to-search-for-great-programmers.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-48964651296773156172014-12-06T06:00:00.000-08:002014-12-06T06:00:00.276-08:00What is really interesting about Quantitative Behavioral Finance<!-- copy and paste this code --><table style="font-size: 11px;" cellspacing="0" cellpading="0"><tr><td colspan="2"><a href="http://farm1.static.flickr.com/188/456398100_c151a4b1e8.jpg" target="_blank"><img src="http://farm1.static.flickr.com/188/456398100_c151a4b1e8.jpg" style="border: 1px solid #ccc;"></a></td></tr><tr><td>photo by <a href="http://www.flickr.com/photos/stuckincustoms/456398100/" target="_blank">Stuck in Customs</a></td><td align="right"> via <a href="http://www.photoree.com" target="_blank">PhotoRee</a></td></tr></table><!-- end of EMBED code-->
<p>
Quantitative behavioral finance has not been with out for a very long time. As a relatively recent development and area of discourse, it has only begun to gain a following. One very interesting aspect of this field is the use of experimental asset markets. These studies are based on experiments conducted on a small group of people (but with real money hence a real market) to examine where rational expectations and classical game theory fails to explain human behavior. This is basically small-scale version of the prediction markets such as the <a href="https://tippie.uiowa.edu/iem/">Iowa Electronic Markets</a>, <a href="https://www.intrade.com">Intrade</a>, and <a href="http://www.betfair.com/">Betfair</a>. However, unlike prediction markets where the ultimate objective is to predict an external event, experimental asset markets are more interested in the mechanics and patterns of the market itself. Caginalp, Vernon Smith (Nobel Memorial Economics Prize recipient of 2002), and David Porter have a <a href="http://www.pitt.edu/~caginalp/experassetmarket.htm">couple of papers on experiments in this mode</a>. Both experiments examine how financial bubbles can happen. Some of the take-aways are that excess cash and information asymmetry due to lack of an open book may exacerbate bubbles. I think the matter of information asymmetry is very salient. Despite all the effort and money invested in improving information infrastructure by banks and hedge funds, ultimately information is distributed non-uniformly to market participants. This is most obvious in the case of the retail investor who neither has the time nor resources to obtain and analyze all the market information.
</p>
<a href="http://www.typedynamic.com/2014/12/what-is-really-interesting-about.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-29918766340299877572014-12-05T07:28:00.001-08:002014-12-05T08:52:26.642-08:00Container Virtualization Options<!-- copy and paste this code --><table style="font-size: 11px;" cellspacing="0" cellpading="0"><tr><td colspan="2"><a href="http://mrg.bz/pVyFUt" target="_blank"><img style="width: 470px;" src="http://cdn.morguefile.com/imageData/public/files/s/sioda/preview/fldr_2008_11_28/file0001748937726.jpg" style="border: 1px solid #ccc;"></a></td></tr><tr><td>photo by <a href="http://morguefile.com/archive/display/88940" target="_blank">sioda</a></td><td align="right"> via <a href="http://www.photoree.com" target="_blank">PhotoRee</a></td></tr></table><!-- end of EMBED code-->
<p>
Looks like the container virtualization space is becoming a little more interesting this week. Previously, Docker was the only more or complete standard container implementation (with definition of image, image creation, and container start/stop management). There was <a href="http://www.ubuntu.com/cloud/tools/lxd">Canonical's LXD</a>, it didn't seem to be garnering nearly as much attention and support since it was only <a href="http://www.flockport.com/canonical-announces-new-lxd-to-create-the-next-big-hypervisor-around-container-technologies/">announced a month ago</a>. However, with the Docker and CoreOS organizations starting to encroach on each other's territory, the CoreOS community has released an early version of their own container runtime, <a href="http://www.infoq.com/news/2014/12/coreos-rocket">Rocket</a>. On the balance, Docker has <a href="http://blog.docker.com/2014/12/docker-announces-orchestration-for-multi-container-distributed-apps/">moved into the container cluster orchestration and management space with Docker Swarm and Docker Compose</a>, the latter being still in the design stage.
</p>
<a href="http://www.typedynamic.com/2014/12/container-virtualization-options.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-49503971061089621272014-12-05T06:00:00.000-08:002014-12-05T06:00:10.654-08:00Containers versus Virtual Machines<p>
Containers-based systems (e.g., Docker, LXC, cgroups) and virtual machines (VMWare, Xen) both seek to bring the benefits of virtualization to the data center and developer workflow. They have considerable overlap in benefits. Although both do some kind of virtualization to enable better utilization of physical hardware, there are also some key differences. Containers do virtualization at the OS kernel-level. Hence isolation is limited to what the kernel can enforce. Containers do a lot of sharing of layers of file systems, courtesy of <a href="http://aufs.sourceforge.net/">AuFS</a>, which potentially makes better use of disk space and image space than virtual machines which commit the entire contents of a VM's disk to the disk image.
</p>
<a href="http://www.typedynamic.com/2014/12/containers-versus-virtual-machines.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-30735974410487825382014-12-04T06:00:00.000-08:002014-12-04T06:00:08.705-08:00Supervisor trees<p>
In my past few posts, I have focused on fault tolerant distributed systems as implemented through cluster managers. Apache Mesos, Kubernetes, and many others all attempt to support fault tolerance by auto-restarting and other self-healing techniques at the cluster manager level. As such, they rightly claim that they are the new operating systems of the cloud. It turns out, however, cluster managers certainly do not have a monopoly on fault tolerance features. Long before Mesos, Kubernetes, and possibly even University of Wisconsin's <a href="http://research.cs.wisc.edu/htcondor/">Condor</a>, a distributed processing system with considerable more pedigree, Erlang had supervisor trees and supervisor behaviors (a kind of language interface) in the runtime thus supporting large, highly fault tolerant distributed systems decades ago.
</p>
<a href="http://www.typedynamic.com/2014/12/supervisor-trees.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-76107727124405483482014-12-03T06:00:00.000-08:002014-12-03T06:00:04.419-08:00Security in Containerization Technology<p>
One lingering worry with containerization is security. Previously, with conventional type 0 and type 1 (native, bare-metal) hypervisor technology, we greatly limited our trusted based to small hypervisors (e.g., Xen is < 150kloc). Some were so small (seL4 core was 7.5kloc) that they were amenable to mechanized formal verification. OSes supporting containers, in contrast, are much larger. Even CoreOS, intended as a slimmed down version of the Chrome OS Linux kernel that just supports modern bare-metal architectures for containers, that is fundamentally more challenging to vet, not to mention verify, than a simple hypervisor. Etcd and fleet alone add up to 44k sloc of Go. So for all the great inroads we were making in verification, the move towards containerization in the data center brings new challenges and potentially resets some of the progress the community has made in mechanically verifying security and functional correctness of the lowest layers of software systems and infrastructure.
</p>
<a href="http://www.typedynamic.com/2014/12/security-in-containerization-technology.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-77318925177687522222014-12-02T06:00:00.000-08:002014-12-02T06:00:01.019-08:00ClusterHQ Flocker <p>
Flocker does multi-host orchestration for Docker containers. It is intended mainly as a means for containerizing and orchestration distributed data stores and databases although in principle it can deploy any app. Unlike some of the other solutions out it, Flocker aims to support checkpointing of stateful and stateless containers to support migration of (running) containers across nodes. This seems like a great feature if one wanted to do work stealing rescheduling of containers as the execution profile changes and other nodes become available. Flocker provides its own NAT layer for mediating communication between containers across nodes. It also supports ZFS persistent volumes to maintain state. Flocker itself does not aim to do any particularly sophisticated scheduling (c.f. Kubernetes) but instead relies on the user to supply scheduling.
</p>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0tag:blogger.com,1999:blog-2740067176225782215.post-47975046501405573272014-12-01T06:00:00.000-08:002014-12-02T19:27:05.682-08:00Kubernetes and Decking Container Cluster Managers<figure>
<img style="width:490px;" src="https://raw.githubusercontent.com/commonlisp/kubernetes/master/docs/architecture.png">
<figcaption>Fig. 1: Kubernetes Architecture</figcaption>
</figure>
<p>
Kubernetes manages user-defined collections of containers called pods. Note that "pod" refers to the running container and not a static image (in Docker terminology). Besides containers, a pod can also have persistent storage attached as a volume and also define custom container health checks. Pods themselves can be organized together into "groups", a kind of "API object", which in turn can be referenced by label. There are two other main forms of API objects: replication controllers and services. The former produces a fixed number of replicas of a pod template. The latter defines internal and external ports for establishing connectivity across pods.
</p>
<a href="http://www.typedynamic.com/2014/12/kubernetes-and-decking-container.html#more">Read more »</a>Georgehttp://www.blogger.com/profile/04534592972924262108noreply@blogger.com0