An Empirical Study of the Reliability of Operating System Utilities

Abstract

Operating system facilities, such as the kernel and utility programs, are assumed to be reliable. Recent experience has led us to question the robustness of our operating system utility programs under unusual input streams. Specifically, we were interested in a basic form of testing: whether the utilities would crash or hang (infinite loop) in response to unusual input. We constructed a set of tools to test this form of reliability. Almost 90 utilities were tested on each of six versions of the UNIX operating system. Included in this list of systems was one that underwent commercial product testing. As a result of our tests, we were able to crash 24-33% of the utilities that we tested, including commonly used utilities such as editors, shells and document formatters. These were programs that either terminated abnormally, generating a core dump, or programs that had infinite loops. We then examined each utility program that crashed and identified the cause. These results were then categorized by the cause of crash. We describe the cause of the crashes, the programming practices that led to the errors, and make some suggestions on how to avoid these problems in the future.