Computing Security and Human Values – Design/Implementation Concerns – The Research Center on Computing & Society

Text-only Version

Peter G. Neumann

6. Design/Implementation Concerns

Various issues need to be considered relating to system design and implementation:

Do the system security requirements properly reflect the social requirements? Often there are glaring omissions.
Are the system security requirements properly enforced by the actual system? There are often flaws in system design and implementation.
What are the intrinsic limitations as to what can and cannot be guaranteed? Nothing can be absolutely guaranteed. There are always possibilities for undetected exceptions. We can always do better, but cannot be perfect. It is desirable to design systems so that if something undesirable does happen, it may be possible to contain it in some sense relevant to the problem, or to undo it, or to compensate for it.
Is the system being used in a fundamentally unsound way that clearly violates or permits violations of the desired behavior? In many cases the absence of guarantees combined with the likelihood of serious negative consequences suggests that such use is fundamentally unsound.

7. Operational Concerns

Even a system that has been ideally designed and implemented can be compromised if it is operationally not soundly administered. Some of the key issues relating to proper administrative management include the following desiderata:

Ability to recognize and eliminate in a timely fashion various system flaws, configuration vulnerabilities, and procedural weaknesses. Such problems tend to remain of little concern until actually exploited in some dramatic way, at which point a little panic often results in a quick fix that solves only a small part of the problems.
Ability to react quickly to evident emergencies, e.g., massive penetrations or other computer system attacks. Preparedness is not a natural instinct in the face of unknown or unperceived threats.
Willingness to communicate the existence of vulnerabilities and ongoing attacks to others who might have similar experiences. In some cases corporate secrecy is important to those who fear negative competitive impacts from disclosures of losses. In other cases there is a lack of community awareness as to the global nature of the problems. Interchange of information can be an enormous aid to good management.
Recognizing potential abuses, e.g., insiders privately selling off sensitive information or ‘fixing’ database entries (e.g., removing outstanding warrants from criminal records) and dealing proactively with them.

8. Antisocial Behaviors

There are various manifestations of antisocial behavior that can be related to computer system design, development, and operation, as well as to specific deviations from ethical, moral, and/or legal behavior.

8.1 ‘Hacking’, Good and Bad

‘Hacker’ was originally a benevolent term, not a pejorative term. In light of media responses to recent system misuses, the negative use seems to have prevailed, and has permanently contaminated the term, more or less preempting its use with respect to benevolent hackers. There are many beneficial consequences of an open society in which free exchange of ideas and programmers is encouraged. However, there will always remain serious potentials for misuse.
Misuse may originate intentionally or accidentally. Both cases represent serious potential problems. (See the next section for a discussion of what to do about these problems.)
Misuse by authorized users and misuse by unauthorized users are both serious potential problems, although in any particular application either one of these problems may be more important than the other. It depends on the environment.
What is actually “authorized” in any given application is often unclear, and may be both poorly defined and poorly understood. This is discussed in the next section.

8.2 Summary of Modes of Misuse

Trap doors and other vulnerabilities represent serious potential sources of security compromise, whether by authorized users or by unauthorized users. Many systems have fundamental security flaws; some flaws can be exploited by people without deep system knowledge, while other flaws cannot.
Misuse of authority by legitimate users is in some system environments more likely than external intrusions (e.g., where there are much more limited opportunities for intrusions because of the absence of dial-up lines and network connections). Such misuse may be done by partially privileged users as well as by omnipotent users, particularly when vulnerabilities are exploited as well. Note that the distinction between authorized and unauthorized users is a very tricky one, as discussed in Section 9.
There are various modes of abusive system contamination, often lumped together under the rubric of pest programs. These include Trojan horses (e.g., time bombs, logic bombs, letter bombs, etc.), human-propagated Trojan horses, self-propagating viruses, malevolent worms, and others. Following the mythology, a Trojan horse is a program (or data or hardware or whatever) that contains something capable of causing an unanticipated and usually undesirable consequence when invoked by an unsuspecting user. The distinctions among the various forms of pest programs tend to cause inordinate philosophical and pseudo-religious arguments among supposedly rational people, but are more or less irrelevant here. So-called personal computer viruses are generally Trojan horse contaminations that are spread inadvertently by human activity. The recent proliferation of old viruses and the continued appearances of new strains of viruses are both phenomena of our times; worse yet, stealth viruses that can hide themselves and in some cases mutate to hinder detection are just beginning to emerge.

8.3 Deleterious Computer-System-Oriented Effects

Losses of confidentiality. Information (e.g., data and programs) may be obtained in a wide variety of ways, including direct acquisition by the obtainer, direct transmittal from a donor, inadvertent access permission from the purveyor or second party, or indirectly. Indirect acquisition includes inferences derived contextually from available information. One form of inference involves the so-called aggregation problem, in which the totality of information is somehow more sensitive than any of the data items taken individually. Another form of indirect acquisition results from the exploitation of a covert channel, which involves a somewhat esoteric signaling through a channel not ordinarily used to convey information, such as the presence or absence of an error message signifying the exhaustion of a shared resource.
Losses of system integrity, application integrity, and system predictability. There are numerous relevant forms of integrity. System programs, data, and control information may be changed improperly. The same is true of user programs, data, and control information. Any such changes may prevent the system from dependably producing the desired results. These are basically notions of internal consistency. External consistency is also a serious problem, for example, if the data in a database is not consistent with the real-world data it purports to represent. Erroneous information can have serious consequences in a variety of contexts.
Denials of service and losses of resource availability. There are deleterious effects that involve neither losses of confidentiality nor losses of integrity. These include serious performance degradations, loss of critical real-time responsiveness, unavailability of data when needed, and other forms of service denial.
Other misuse. The above list is far from complete, as there are many further types of misuse. For example, misuse may involve undetected thefts of services (e.g., computing time) or questionable applications (e.g., running private businesses from employers’ facilities).

8.4 Social Consequences

Violation of privacy and related human rights, (e.g., constitutional). Loss of confidentiality can clearly result in serious privacy problems, whether intentionally or unintentionally caused. All of the above modalities of loss of confidentiality can have serious consequences. Furthermore, the effects of erroneous information can be even more serious, in the senses of both internal and external consistency.
Software piracy. Theft of programs, data, documentation, and other information can result in loss of revenues, loss of recognition, loss of control, loss of responsibility without loss of liability, loss of accountability, and other serious consequences.
Effects on human safety. Misuse of a life-critical system can result in deaths and injuries, whether it is done accidentally or intentionally.
Legal issues. The potential legal effects are quite varied. There can be law suits against misusers, innocent users, and system purveyors. Some of those lawsuits would undoubtedly be frivolous or misguided, but nevertheless causing considerable agony to the accused. Computer “crimes” have already been a source of real difficulties for law enforcement communities, as well as for both guilty and innocent defendants.
Perceptions. Increased interconnectivity, inter-communicability, and use of shared resources are clearly desirable goals. However, fears of Trojan horses, viruses, losses of privacy, theft of services, etc. are likely to create a community that is either paranoid or oblivious to and vulnerable to the social dangers.

Back to the top

Go to: 9. System Considerations

Home > Research Resources > Computer Security > Computer Security and Human Values

The Research Center on Computing & Society
at Southern Connecticut State University
501 Crescent Street • New Haven, CT 06515
Director: (203) 392-6790 • e-mail: webmaster@computerethics.org

© 2000 – 2007 – Research Center on Computing & Society