Updated: 07 June 2001

Volume Shadowing for OpenVMS

Volume Shadowing for OpenVMS is designed primarily to be a data availability product and not a performance product. Recognizing that the topics of performance and data availability cannot be completely separated from each other, this chapter discusses the performance effects that can result on systems using Volume Shadowing for OpenVMS. Several factors affect the performance of a shadow set, including the following:

• I/O access path (local versus remote)
• Size of I/O requests
• Data access patterns (random or sequential)
• Read-to-write ratio
• Shadow set configuration
• State of a shadow set (steady or transient)
• Whether or not you use the shadowing copy and merge performance assists (see Section 8.2.1).
• Other work load on the system utilizing common resources (CPUs, disks, controllers, interconnects)
• Striping (RAID) implementation

The following sections examine how the state of a shadow set and its configuration can affect resource utilization and performance. Some guidelines for controlling the use of system resources are also provided in Section 8.3. Because there is no significant difference in the performance of a nonshadowed disk and a one-member shadow set, all discussions that follow apply to multiple-member shadow sets.

8.1 Performance During Steady State

A shadow set is in a steady state when all of its members are consistent and no copy operation or merge operation is in progress. Overall, the performance of a shadow set in a steady state compares favorably with that of a nonshadowed disk. Read and write I/O requests processed by a shadow set utilize a very small amount of extra CPU processing time as compared with a nonshadowed disk. A shadow set often can process read requests more efficiently than can a nonshadowed disk because it can use the additional heads to respond to multiple read requests simultaneously.

For a shadow set in a steady state, the shadowing software handles read and write operations in the following manner:

• Write I/O requests are issued concurrently to all members of the shadow set. Because each member must be updated before the I/O request is considered complete, the overall completion time for a write operation is determined by the member unit with the longest access time from the node issuing the write request. Depending on how the shadow set is configured and the access paths to the individual member units, you might observe a slight increase in the time it takes to complete write I/O requests. The steady state performance is generally better to a member that is locally connected because the access path is shorter and more direct than the access path to a served member. For example, you might notice degraded write performance on shadow sets that include some members that are accessed through an MSCP server across an Ethernet, where each member is locally connected to a separate node.
• Read I/O requests are issued to only one member unit. Volume Shadowing for OpenVMS attempts to access the member unit that can provide the best completion time. In terms of I/O throughput, a two-member shadow set can satisfy nearly twice as many read requests as a nonshadowed disk (and even more throughput is possible with a three-member shadow set). The shadow set can use the additional disk read heads to respond to multiple read requests at the same time. Thus, a steady-state shadow set can provide better performance when an application or user reads data from the disk. However, the performance gains occur mainly when the read requests queued to the shadow set come in batches such that there are as many read requests as there are member units.

Even though the read performance of a shadow set in steady-state has the potential for better performance, the primary purpose of volume shadowing is to provide data availability. It is not appropriate to use volume shadowing as a means to increase the read I/O throughput of your applications (by explicitly increasing the I/O work load). This is because the same level of performance cannot be expected during situations when copy or merge operations must take place to add new members or preserve data consistency, or when members are removed from the shadow set. Section 8.2 discusses performance considerations when the shadow set is in a transient state.

8.2 Performance During Copy and Merge Operations

A shadow set is in a transient state when some or all of its members are undergoing a copy or merge operation. During merge operations, Volume Shadowing for OpenVMS ensures consistency by reading the data from one member and making sure it is the same as the data contained in the same LBNs on the other members of the shadow set. If the data is different, the shadowing software updates the LBN on all members. For copy operations, the shadowing software reads data from a source member and writes the data to the same LBN on target members.

At the same time it performs a merge or copy operation, the shadowing software continues to process application and user I/O requests. The I/O processing necessitated by a copy operation can result in decreased performance as compared with the possible performance of the same shadow set under steady-state conditions. However, if your shadow set members are configured on controllers that support the shadowing assisted copy and assisted merge operations, you can significantly improve the speed with which a shadow set performs a copy or merge operation. Volume Shadowing for OpenVMS supports both assisted and unassisted merge and copy operations.

The following list describes how the performance of a shadow set might be affected while an unassisted merge or copy operation is in progress. See Chapter 6 for a description of assisted copy and merge operations.

• Copy operations
  A copy operation is started on a two- or three-member shadow set either when you mount the shadow set to create it or to add a new member to an existing shadow set. During a copy operation, members that are targets of the operation cannot provide data availability until the operation completes. Therefore, the shadowing software performs the copy operation as quickly as possible to make the shadow set fully available.
  During a copy operation, the shadowing software gives equal priority to user and application I/O requests and to I/O requests necessary to complete the copy operation. The performance of a shadow set during a copy operation is reduced because:
  • The shadowing software must follow special protocols for user read and write I/O requests during a copy operation.
  • Copy operation I/O requests are large in size and have the same priority as user and application I/O requests.
  
  In addition, other system resources are utilized during a copy operation. Depending on the access path to the individual shadow set members, these resources could include the disk controller, interconnect adapters such as the CI (computer interconnect) or Ethernet adapters, CPUs, and interconnects.
  Because you explicitly start copy operations when you mount a new shadow set or add members to an existing set, you can control when the shadowing software performs a copy operation. Therefore, you can minimize the effect on users and applications in the system by limiting the number of copy operations that occur at the same time. For example, when you create new sets or add new members, try to add the sets or members during periods of low system activity, and do not mount several sets at the same time.
  For FDDI OpenVMS Cluster systems, performance improves if the MOUNT command is entered on the node closest to the disk that will be the target of the copy operation. A local MOUNT command is more efficient because when the target disk is MSCP served to the system managing the copy operation, the write operation takes a small amount of additional time to write each different block found.
• Merge operations
  In contrast to copy operations, merge operations are not under the control of a user or program. The shadowing software automatically initiates a merge operation on a shadow set as a result of the failure of a node on which the shadow set is mounted.
  As in the case of a copy operation, the volume shadowing software ensures that all I/O requests to the shadow set follow appropriate protocols to ensure data consistency. However, when a shadow set is undergoing a merge operation, full data availability exists in the sense that individual members of the set are valid data sources and are accessible by applications and users on the system. Therefore, it is not urgent for the shadowing software to finish the merge operation, especially when the system is being heavily used. Because of this major distinction from a copy operation, the shadowing software implicitly places a higher priority on user activity to the shadow set. Volume Shadowing for OpenVMS does this by detecting and evaluating system load, and then dynamically controlling or throttling the merge operation so that other I/O activity can proceed without interference.
  Because the merge throttle slows merge operations when there is heavy application and user I/O activity on the system, the merge operation can take longer than copy operations. The merge throttle allows application and user activity to continue unimpeded by the merge operation when heavy work loads are detected. The automatic throttle also prevents resource saturation by the merge operation on shared components such as CI or Ethernet interconnects in the system.
  On the other hand, the read performance of a shadow set during a merge operation might seem reduced because the shadowing software must perform data integrity checks on all members for every read request. The volume shadowing software reads the data from the same LBN on all members of the shadow set, compares the data, and repairs any inconsistencies before returning the read data to the user.

There are two types of performance assist: the merge assist and the copy assist. The merge assist improves performance by using information that is maintained in controller-based write logs to merge only the data that is inconsistent across a shadow set. When a merge operation is assisted by the write logs, it is referred to as a minimerge. The copy assist reduces system resource usage and copy times by enabling a direct disk-to-disk transfer of data without going through host node memory.

Assisted merge operations are usually too short to be noticeable. Improved performance is possible during the assisted copy operation because it consumes less CPU and interconnect resources. Although the primary purpose of the performance assists is to reduce the system resources required to perform a copy or merge operation, in some circumstances you may also observe improved read and write I/O performance.

Volume Shadowing for OpenVMS supports both assisted and unassisted shadow sets in the same OpenVMS Cluster configuration. Whenever you create a shadow set, add members to an existing shadow set, or boot a system, the shadowing software reevaluates each device in the changed configuration to determine whether it is capable of supporting either the copy assist or the minimerge. Enhanced performance is possible only as long as all shadow set members are configured on controllers that support performance assist capabilities. If any shadow set member is connected to a controller without these capabilities, the shadowing software disables the performance assist for the shadow set.

When the correct revision levels of software are installed, the copy assist and minimerge are enabled by default, and are fully managed by the shadowing software.

8.2.2 Effects on Performance

The copy assist and minimerge are designed to reduce the time needed to do copy and merge operations. In fact, you may notice significant time reductions on systems that have little or no user I/O occurring during the assisted copy or merge operation. Data availability is also improved because copy operations quickly make data consistent across the shadow set.

Minimerge Performance Improvements

The minimerge feature provides a significant reduction in the time needed to perform merge operations. By using controller-based write logs, it is possible to avoid the total volume scan required by earlier merge algorithms and to merge only those areas of the shadow set where write activity is known to be in progress.

Unassisted merge operations often take several hours, depending on user I/O rates. Minimerge operations typically complete in a few seconds and are usually undetectable by users.

The exact time taken to complete a minimerge depends on the amount of outstanding write activity to the shadow set when the merge process is initiated, and on the number of shadow set members undergoing a minimerge simultaneously. Even under the heaviest write activity, a minimerge should complete in less than 1 minute. Additionally, minimerge operations consume minimal compute and I/O bandwidth.

Copy Assist Performance Improvements

Table 8-1 shows the approximate time required to perform copy operations on shadow sets that consist of two RA82 disks connected to an HSC70 controller on a VAX 8700 computer. The table shows assisted and unassisted copy times measured for one, two, three, and four concurrent copy operations, where all source and target disks are on separate HSC requesters.

Note that the copy times in Table 8-1 reflect the best possible performance measurements taken on a controlled test system with no user I/O load. Copy times vary according to each configuration and generally take longer on systems supporting user I/O. Performance benefits are also enhanced by having the source and target disks on different HSC requesters.

Table 8-1 Copy Times (In Minutes) for Two-Member RA82 Shadow Sets

Type of Shadowing	1 Set	2 Sets	3 Sets	4 Sets
Phase II: Assisted copy	9	17	25	34
Phase II: Unassisted copy with identical data	24	28	34	41
Phase II: Unassisted Copy with different data	91	120	157	200

Table 8-1 shows that, for a single copy operation, a phase II assisted copy is significantly faster than any other. Note that, as the number of simultaneous copies grows, the timing proportions change. This is caused by differing levels of parallelism in the various copying algorithms. Also, the overall copy times and their proportions differ with the disk type because each disk varies in total blocks and data transfer speed.

Additionally, Table 8-1 shows that unassisted phase II copies vary significantly in time, based on the similarity of data on the source and target disks. To make blocks with different data consistent, extra I/O operations are required. Thus, unassisted copy operations take significantly longer to make a shadow set consistent when the disk members contain different data. The phase II assisted copy operation does not depend on the similarity of data on the disks.

In addition, the following subsection describes how you can control the number of assisted copies an HSC controller can perform.

Determining the DCD Connection Limit

The default DCD connection limit of 4 is based on measurements obtained from tests of simultaneous assisted copies on several types of disks. In most cases, if more than four simultaneous copies are required, lower copy times are achieved when the additional copies are unassisted. For example, if six copies are necessary, four are assisted and two are unassisted. The default setting assumes that any unassisted copies will be performed on disks with nearly identical data.

You may need to increase the DCD connection limit under the following circumstances:

• Copy operations are generally performed on disks with dissimilar data. In this case, unassisted copies take significantly longer than assisted copies. Therefore, setting a higher DCD connection limit would be advantageous.
• Good user I/O performance is more important than total elapsed time for the copy operations. Assisted copy operations conserve system I/O resources, thereby permitting good user I/O performance.

The best mix of assisted and unassisted copies can vary based on the CPU, the HSC, and the disks involved because each environment has a different performance profile.

Setting the Number of Copy Assists Per HSC

You can use the HSC command SET SERVER DISK/DCD_CONNECTION_LIMIT to control the mix of assisted and unassisted copies performed by a given HSC.

Follow these steps for each HSC controller on which you want to change the DCD connection limit.

1. Press Ctrl/C to get to the HSC> prompt.
2. At the HSC prompt, enter the following commands to set the DCD connection limit to 6:

   HSC> RUN SETSHO SETSHO> SET SERVER DISK/DCD_CONNECTION_LIMIT=6 SETSHO>

Sections 8.1 and 8.2 describe the performance impacts on a shadow set in steady state and while a copy or merge operation is in progress. In general, performance during steady state compares with that of a nonshadowed disk. Performance is affected when a copy or a merge operation is in progress to a shadow set. In the case of copy operations, you control when the operations are performed.

However, merge operations are not started because of user or program actions. They are started automatically when a CPU fails. In this case, the shadowing software reduces the utilization of system resources and the effects on user activity by throttling itself dynamically. Minimerge operations consume few resources and complete rapidly with little or no effect on user activity.

The actual resources that are utilized during a copy or merge operation depend on the access path to the member units of a shadow set, which in turn depends on the way the shadow set is configured. By far, the resources that are consumed most during both operations are the adapter and interconnect I/O bandwidths.

You can control resource utilization by setting the SHADOW_MAX_COPY system parameter to an appropriate value on a CPU based on the type of CPU and the adapters on the machine. SHADOW_MAX_COPY is a dynamic system parameter that controls the number of concurrent copy or merge threads that can be active on a single CPU. If the number of copy threads that start up on a particular CPU is more than the value of the SHADOW_MAX_COPY parameter on that CPU, only the number of threads specified by SHADOW_MAX_COPY will be allowed to proceed. The other copy threads are stalled until one of the active copy threads completes.

For example, assume that the SHADOW_MAX_COPY parameter is set to 3. If you mount four shadow sets that all need a copy operation, only three of the copy operations can proceed; the fourth copy operation must wait until one of the first three operations completes. Because copy operations use I/O bandwidth, this parameter provides a way to limit the number of concurrent copy operations and avoid saturating interconnects or adapters in the system. The value of SHADOW_MAX_COPY can range from 0 to 200. The default value is 4.

Chapter 3 explains how to set the SHADOW_MAX_COPY parameter. Keep in mind that, once you arrive at a good value for the parameter on a node, you should also reflect this change by editing the MODPARAMS.DAT file so that when invoking AUTOGEN, the changed value takes effect.

In addition to setting the SHADOW_MAX_COPY parameter, the following list provides some general guidelines to control resource utilization and the effects on system performance when shadow sets are in transient states.

• Create or add members to shadow sets when your system is lightly loaded.
• The amount of data that a system can transfer during copy operations varies depending on the type of disks, interconnect, controller, the number of units in the shadow set, and the shadow set configuration on the system. For example, approximately 5% to 15% of the Ethernet or CI bandwidth might be consumed for each copy operation (for disks typically configured in CI or Ethernet environments).
• While creating unassisted three-member shadow sets, add all members at the same time in a single mount command rather than in two separate mount commands. Adding several members at once optimizes the copy operations by starting a single copy thread that reads from the source members once, and performs write I/O requests to the target members in parallel.
• For satellite nodes in a mixed-interconnect or local arean OpenVMS Cluster system, set the system parameter SHADOW_MAX_COPY to a value of 0 for nodes that do not have local disks as shadow set members.
• Do not use the MOUNT/CLUSTER command to mount every shadow set across the cluster unless all nodes must access the set. Instead, use the MOUNT/SYSTEM command to mount the shadow sets on only those nodes that need to access a particular set. This reduces the chances of a shadow set going into a merge state. Because a shadow set goes into a merge state only when a node that has the set mounted fails, you can reduce the chances of this happening by limiting the number of nodes that mount a shadow set, especially when there is no need for a node to access the shadow sets.
• Because a copy operation can occur only on nodes that have the shadow set mounted, create and mount shadow sets on the nodes that are local (have direct access) to the shadow set members. This allows the copy threads to run on these nodes, resulting in faster copy operations with fewer resources utilized.
• If you have shadow sets configured across nodes that are accessed through the MSCP server, you might need to increase the value of the MSCP_BUFFER system parameter in order to avoid I/O fragmentation. You can determine whether you need to increase the MSCP_BUFFER parameter by running AUTOGEN with FEEDBACK specified for the served nodes. You can use the feedback reports to observe how efficient the current setting of MSCP_BUFFER is.
• Dual-pathed and dual-ported shadowed disks in a OpenVMS Cluster system can provide additional coverage against the failure of nodes that are directly connected to shadowed disks. This type of configuration provides higher data availability with reasonable performance characteristics.
• Use the preferred path option to ensure dual-ported drives are accessed via the same controller so that the shadowing software will perform assisted copy operations.

Copyright © Compaq Computer Corporation 1998. All rights reserved. Legal

5423PRO_009.HTML