A common design pattern in fault-tolerant distributed systems is the retry pattern. A given operation may experience a variety of failures:
rare transient failures
(e.g., corrupted packet) can be recovered from immediately and thus should retry immediately common transient failures
(e.g., network busy) can retry after waiting for a period of time (possibly with exponential backoffpermanent failure
should not retry, bail out and clean up
Of course, the final case is that the operation succeeds and the function must do some work to address that. This is an interesting design pattern not only for distributed systems but also fault-tolerant systems in general. For example, high-performance JavaScript engines have parsers and tokenizers that must be robust to various failures. In fact, it is one example where large systems have used multiple exit points, more complicated control flow for which C-based programs may use gotos.
