OpenVMS Guide to System Security

Updated: 24 May 2001

Order Number: AA--Q2HLD--TE

January 1999

This guide describes the security features available through the OpenVMS operating system. It explains the purpose and proper application of each feature in the context of specific security needs.

Revision/Update Information: This manual supersedes the OpenVMS Guide to System Security, Version 7.1

Software Version: OpenVMS Alpha Version 7.2
OpenVMS VAX Version 7.2

Compaq Computer Corporation
Houston, Texas

January 1999

Compaq Computer Corporation makes no representations that the use of its products in the manner described in this publication will not infringe on existing or future patent rights, nor do the descriptions contained in this publication imply the granting of licenses to make, use, or sell equipment or software in accordance with the description.

Possession, use, or copying of the software described in this publication is authorized only pursuant to a valid written license from Compaq or an authorized sublicensor.

Compaq conducts its business in a manner that conserves the environment and protects the safety and health of its employees, customers, and the community.

The following are trademarks of Compaq Computer Corporation: Alpha, Compaq, DECdtm, DECdirect, DIGITAL, OpenVMS, VAX, VAX DOCUMENT, VAXcluster, VMS, and the Compaq logo.

The following are third-party trademarks:

Display POSTSCRIPT is a registered trademark of Adobe Systems Incorporated.

All other trademarks and registered trademarks are the property of their respective holders.

ZK6346

The OpenVMS documentation set is available on CD-ROM.

This document was prepared using VAX DOCUMENT, Version V3.2n.

Contents Index

Preface

Intended Audience

This guide is designed for users and for administrators responsible for protecting operating systems from tampering, observation, or theft of services by unauthorized users. The term security administrator is used in this guide to refer to the person or persons responsible for system security.

Document Structure

This guide contains the following information:

Part 1: Overview
Gives security administrators an overview of security issues, conceptual design features, and security features specific to OpenVMS systems.
- Chapter 1 discusses levels of security requirements and describes three sources of security failures.
- Chapter 2 introduces the reference monitor concept of security design and provides an overview of the operating system's security features.
Part 2: Security for the User
Describes security actions and features for the general user.
- Chapter 3 provides information for the general user about the login and logout processes and the responsible use of passwords.
- Chapter 4 and Chapter 5 describe object protection features in detail.
Part 3: Security for the System Administrator
Describes security actions and features for the security administrator.
- Chapter 6 describes the general tasks of a security administrator.
- Chapter 7 describes methods of controlling system access.
- Chapter 8 describes methods of controlling access to system data and resources.
- Chapter 9 describes security-auditing features.
- Chapter 10 describes how to recognize when a system is under attack and how to protect and defend your system.
- Chapter 11 describes security-related actions specific to clustered systems, such as setting up common system files and synchronizing authorization data.
- Chapter 12 describes security considerations for systems using networking.
- Chapter 13 describes how to set up and manage protected subsystems.
- Appendix A provides a summary of all the user privileges available on the operating system and describes who may need them.
- Appendix B lists the protection codes and ownership that Compaq provides for critical system files.
- Appendix C describes how to operate OpenVMS systems in a Division C, Class 2 (C2) security environment.
- Appendix D provides examples of security alarm messages.
The Glossary provides definitions of security-related terms introduced in this guide.

Reader's Comments

Compaq welcomes your comments on this manual.

Print or edit the online form SYS$HELP:OPENVMSDOC_COMMENTS.TXT and send us your comments by:

Internet openvmsdoc@zko.mts.dec.com

Fax 603 884-0120, Attention: OSSG Documentation, ZKO3-4/U08

Mail Compaq Computer Corporation
OSSG Documentation Group, ZKO3-4/U08
110 Spit Brook Rd.
Nashua, NH 03062-2698

How To Order Additional Documentation

Use the following World Wide Web address to order additional documentation:

http://www.openvms.digital.com:81/

If you need help deciding which documentation best meets your needs, call 800-DIGITAL (800-344-4825).

Conventions

The name of the OpenVMS AXP operating system has been changed to the OpenVMS Alpha operating system. Any references to OpenVMS AXP or AXP are synonymous with OpenVMS Alpha or Alpha.

VMScluster systems are now referred to as OpenVMS Cluster systems. Unless otherwise specified, references to OpenVMS Clusters or clusters in this document are synonymous with VMSclusters.

In this manual, every use of DECwindows and DECwindows Motif refers to DECwindows Motif for OpenVMS software.

The parts of this guide are intended for different audiences, and the meaning of you differs accordingly:

Part 1 addresses primarily security administrators although all users may find the concepts useful.
Part 2 addresses general users.
Part 3 addresses security administrators.

The following conventions are also used in this manual:

Ctrl/ x A sequence such as Ctrl/ x indicates that you must hold down the key labeled Ctrl while you press another key or a pointing device button.

[Return] In examples, a key name enclosed in a box indicates that you press a key on the keyboard. (In text, a key name is not enclosed in a box.)

... Horizontal ellipsis points in examples indicate one of the following possibilities:

Additional optional arguments in a statement have been omitted.
The preceding item or items can be repeated one or more times.
Additional parameters, values, or other information can be entered.

.
.
. Vertical ellipsis points indicate the omission of items from a code example or command format; the items are omitted because they are not important to the topic being discussed.

( ) In command format descriptions, parentheses indicate that, if you choose more than one option, you must enclose the choices in parentheses.

[ ] In command format descriptions, brackets indicate optional elements. You can choose one, none, or all of the options. (Brackets are not optional, however, in the syntax of a directory name in an OpenVMS file specification or in the syntax of a substring specification in an assignment statement.)

{ } In command format descriptions, braces surround a required choice of options; you must choose one of the options listed.

bold text This text style represents the introduction of a new term or the name of an argument, an attribute, or a reason.

italic text Italic text emphasizes important information and indicates complete titles of manuals and variables. Variables include information that varies in system messages (Internal error number), in command lines (/PRODUCER= name), and in command parameters in text (where device-name contains up to five alphanumeric characters).

UPPERCASE TEXT Uppercase text indicates a command, the name of a routine, the name of a file, or the abbreviation for a system privilege.

- A hyphen in code examples indicates that additional arguments to the request are provided on the line that follows.

numbers All numbers in text are assumed to be decimal unless otherwise noted. Nondecimal radixes---binary, octal, or hexadecimal---are explicitly indicated.

Part I
Security Overview

The chapters in this part discuss the following topics:

Sources of security failures ( Section 1.1)
Levels of security requirements ( Section 1.2)
Reference monitor concept of security design ( Section 2.1)
Security features of the operating system ( Section 2.2)

Chapter 1
Understanding System Security

Effective operating system security measures help prevent unauthorized access and theft of computer time and any kind of sensitive information, such as marketing plans, formulas, or proprietary software. These measures can also protect equipment, software, and files from damage caused by tampering.

This chapter provides security administrators with an overview of security measures available with the operating system. Part 3 provides more specific information regarding site security policies, the tasks of security administrators, and methods of protecting site computer systems and resources.

1.1 Types of Computer Security Problems

On any system there can be two types of users: authorized and unauthorized. Any person authorized to use the computer system has the right to access the system and its resources according to the authorization criteria set up by the site security administrator. Usage criteria may include the time of day, types of logins, use of different resources like printers and terminals, and so on. Unauthorized users have no right to use the system at all or only at a given time of day, or they have no right to use certain system resources.

On a computer system, security breaches usually result from one of four types of actions:

User irresponsibility refers to situations where the user purposely or accidentally causes some noticeable damage. One example would be a user who is authorized to access certain files making a copy of a key file to sell.
There is little that an operating system can do to protect sites from this source of security failure. The problem frequently lies in application design deficiencies or inconsistent use of available controls by users and the security administrator. Sometimes the failure to enforce adequate environmental security unwittingly encourages this type of security problem.
Even the best security system will fail if implemented inconsistently. This, along with the failure to motivate your users to observe good security practices, will make your system vulnerable to security failures caused by user irresponsibility. Chapter 3 discusses what users can do to help maintain system security.
User probing refers to situations where a user exploits insufficiently protected parts of the system. Some users consider gaining access to a forbidden system area as an intellectual challenge, playing a game of user versus system. Although intentions may be harmless, theft of services is a crime. Users with more serious intent may seek confidential information, attempt embezzlement, or even destroy data by probing. Always treat user probing seriously.
The system provides many security features to combat user probing. Based on security needs, the security administrator implements features on either a temporary or permanent basis. See Chapter 4 for information on protecting data and resources with protection codes and access control lists.
User penetration refers to situations where the user breaks through security controls to gain access to the system. While the system has security features that make penetration extremely difficult, it is impossible to make any operating system completely impenetrable.
A user who succeeds in penetrating a system is both skilled and malicious. Thus, penetration is the most serious and potentially dangerous type of security breach. With proper implementation of the OpenVMS security features, however, it is also the rarest security breach, requiring unusual skills and perseverance.
Social engineering refers to situations in which an intruder gains access to a system not by technical means, but by deceiving users, operators, or administrators. Potential intruders may impersonate authorized users over the phone. Potential intruders may request information that gains them access to the system, such as telephone numbers or passwords, or they may request an unwitting operator to perform some action that compromises the security of the system.
As the technical security features of operating systems have strengthened in recent years, social engineering has been a factor in a growing percentage of security incidents. Operator training, administrative procedures, and user awareness are all critical factors to ensure that access is not inadvertantly granted to unauthorized persons.

The following chapters explain how to avoid these problems:

Chapter 8 explains how to augment the protection of system files and resources.
Chapter 7 describes the intrusion detection system and how to set its parameters.
Chapter 9 explains how to monitor system activity and be notified by malicious activity.
Chapter 10 suggests how to handle system intrusions.
Chapter 3 and Chapter 6 list topics to include in your site training programs.

1.2 Levels of Security Requirements

Each site has unique security requirements. Some sites require only limited measures because they are able to tolerate some forms of unauthorized access with little adverse effect. At the other extreme are those sites that cannot tolerate even the slightest probing, such as strategic military defense centers. In between are many commercial sites, such as banks.

While there are many considerations in determining your security needs, the questions in Table 1-1 can get you started. Your answers can help determine the levels of your security needs. Also refer to Section 6.2 for a more specific example of site security requirements.

Table 1-1 Event Tolerance as a Measure of Security Requirements
Question: Could you tolerate
the following event? Level of Security Requirements Based on Toleration Responses

Low Medium High

A user knowing the images being
executed on your system
Y
Y
N

A user knowing the names of
another user's files
Y
Y
N

A user accessing the file of another
user in the group
Y
Y
N

An outsider knowing the name of the
system just dialed into
Y
Y
N

A user copying files of other
users
Y
N
N

A user reading another user's
electronic mail
Y
N
N

A user writing data into another
user's file
Y
N
N

A user deleting another user's
file
Y
N
N

A user being able to read
sections of a disk that might
contain various old files

Y

N

N

A user consuming machine time
and resources to perform
unrelated or unauthorized work,
possibly even playing games

Y

N

N

If you can tolerate most of the events listed, your security requirements are quite low. If your answers are mixed, your requirements are in the medium to high range. Generally, those sites that are most intolerant to the listed events have very high levels of security requirements.

When you review your site's security needs, do not confuse a weakness in site operations or recovery procedures as a security problem. Ensure that your operations policies are effective and consistent before evaluating your system security requirements.

1.3 Building a Secure System Environment

There are two sources of security problems outside the operating system domain: employee carelessness and facility vulnerability. If you have a careless or malicious employee or your facility is insecure, none of the security measures discussed in this guide will protect you from security breaches.

Most system penetration occurs through these environmental weaknesses. It is much easier to physically remove a small reel of tape than it is to break access protection codes or change file protection.

Compaq strongly encourages you to stress environmental considerations as well as operating system protection when reviewing site security.

This book discusses operating system security measures. When deciding which of these measures to implement, it is important for you to assess site security needs realistically. While instituting adequate security for your site is essential, instituting more security than actually necessary is costly and time-consuming.

When deciding which security measures to apply to your system, remember the following:

The most secure system is also the most difficult to use.
Increasing security can increase costs in terms of slower access to data, slower machine operations, and slower system performance.
More security measures require more personnel time.

The operating system provides the basic mechanisms to control access to the system and its data. It also provides monitoring tools to ensure that access is restricted to authorized users. However, many computer crimes are committed by authorized users with no violation of the operating system's security controls.

Therefore, the security of your operation depends on how you apply these security features and how you control your employees and your site. By first building appropriate supervisory controls into your application and designing your application with the goal of minimizing opportunities for abuse, you can then implement operating system and site security features and produce a less vulnerable environment. For an example of one organization's security plan, see Chapter 6.

If you require your system to meet the United States government rating of a C2 secure operating system, please refer to Appendix C in this manual.

If you need a higher level of computer security for your OpenVMS secure system, Compaq offers SEVMS, which is the security enhanced version of OpenVMS that provides mandatory access controls to enforce a system-wide security policy.

SEVMS is a U.S. Department of Defense B1-rated secure operating system.

Contents

Index