Saturday, February 11, 2006

Veritas VM Stuff

Veritas VM Basic Setup Tasks

It is assumed that the Veritas software has been installed and that
rootdg has been setup by the installer with a minimum of one disk. If
not, install the software now.

Use the "format" command to display available disks. If necessary,
add new disks to the system and run "drvconfig; devlinks; disks" to
configure the new disks. Do not partition disks to be used by Veritas.

To display disks setup for Veritas, use the command "vxdisk list".

To add more disks to the control of Veritas, use the following command:

The would be c2t0d0, for example, without the slice suffix. The
vxdiskadd script will prompt for necessary information such as the
name of the disk group, whether or not to initialize the new disk,
etc. Rerun the command "vxdisk list" to view the new disk.

To designate an unused disk as a hot-spare disk, use the following command:
vxedit set spare=on

The would be the Veritas name for the disk, such as disk01. Use
"vxdisk list" to confirm that the disk is now a dedicated hot-spare
disk. To remove the disk from the dedicated hot-spare pool, use the
following command:
vxedit set spare=off

By default, Veritas will name disks based on the name of the disk
group. To rename a disk (for example, from disk01 to disk02), use the
following command:
vxedit rename

To place a disk offline (for example, if the disk has become
corrupted), run the following command:
vxdisk offline

The would be c2t0d0, for example, without the slice suffix.

To remove a disk from the diskgroup to which it belongs, run the
following command:
vxdg [-g diskgroup name] rmdisk

To then remove that disk from Veritas control entirely, run the
following command:
vxdisk rm

The same set of commands may be used to move disks from one
diskgroup to another:
vxdg rmdisk
vxdg -g adddisk =

(For example, "vxdg -g testdg adddisk testdg01=c2t0d0".)

Alternatively, you can simply remove the disk from Veritas control
and then reinitialize it with "vxdiskadd". Use "vxdisk list" to
confirm the new affiliation.

Every disk group requires a minimum of one disk. The simplest way to
add a new diskgroup is to run the "vxdiskadd" command on an
unconfigured disk, and when prompted for the name of the diskgroup to
use, enter the desired name of the new diskgroup.

To display information on disk groups, run "vxdg list" or "vxdg list".

To destroy a diskgroup and reuse the disks for other purposes, run
the following command:
vxdg destroy

Failing or failed disks can be removed with the "vxdiskadm" menu
command. To remove a failing disk, select choice 4, "remove a disk for
replacement". To replace the disk later, select choice 5, "replace a
failed or removed disk".

Volume Tips and Tasks
The basic command used to create volumes is "vxassist". The format
is "vxassist make". The length can be specified in units of kilobytes,
megabytes, or gigabytes with the suffixes k, m, or g. By default, new
volumes are created from rootdg. Use "-g" to specify a different

To determine the maximum size that a new volume could be given
particular attributes, use the "vxassist maxsize" command. For
example, to determine the maximum size of a striped volume using
disk01 and disk02:
vxassist maxsize layout=stripe disk01 disk02

Then create the volume using that length. For example: vxassist
make vol01 20g layout=stripe disk01 disk02

By default, the "vxassist" command creates concatenated volumes. For
example, the command "vxassist make newvol 10m" would create a
concatenated volume with the length of 10 MB. To specify the disk(s)
to use, add the disk name(s). For example, "vxassist make newvol 10m

To create striped volumes, add the attribute "layout=stripe". For
exampke, "vxassist make newvol 10g layout=stripe disk01 disk02".

To create RAID-5 volumes, add the attribute "layout=raid5". For
example, "vxassist make newvol 30g layout=raid5".

To create mirrored volumes, add the attribute "layout=mirror". For
example, "vxassist make newvol 10g layout=mirror".

The "vxassist" command can be used to grow or shrink the size of
volumes. Do not shrink the size of volumes below the size of the
filesystem. To grow volumes, run one of the following commands:
vxassist growto
vxassist growby

Similarly, you can shrink the size of volumes:
vxassist shrinkto
vxassist shrinkby

To remove an unneeded volume, unmount any filesystems on the
volume, remove any references to the volume from /etc/vfstab, stop the
volume with "vxvol stop", and then run "vxedit -rf rm" or "vxassist
remove volume".

Existing volumes can be mirrored to add redundancy. To mirror
existing volumes, use the following command:
vxassist mirror

To remove a mirror (more precisely, to remove one of the plexes from
the volume), use the following command:
vxplex -o rm

To stop volumes, use the command "vxvol stop". To stop all volumes,
run the command "vxvol stopall". To restart volumes, use the command
"vxrecover -s" or use the command Veritas will automatically allocate
the necessary space from unused disks. The process may take some time
depending on the size of the volume. When fully synchronized, the
status of the snapshot mirror will change to SNAPDONE. Use the
following command to monitor the progress of the snapshot:
vxtask -l list

2. Detach the snapshot and create the new read-only snapshot volume
with the following command:
vxassist snapshot

The period required to create the snapshot volume is shorter but
it is preferred to limit I/O to the volume during the period required.

3. Clean the contents of the new snapshot volume with the following command:
fsck -y /dev/vx/rdsk/

4. Mount the snapshot volume, perform the backup, unmount the
volume, and then remove the snapshot volume to save space with the
following command: vxedit -rf rm

Disk Group Tips
Diskgroups can be moved from one system to another system. On the
original system, unmount any volumes in the disk group to be deported
and stop them with the command "vxvol stop". Then deport the disk
group with the command "vxdg deport".

Move the disks physically to the new system and either reboot the
system so that Veritas recognizes the new disks, or use the command
"vxdctl enable" to restart vxconfigd.

Import the disks with the command "vxdg import". Then restart all
volumes in the disk group with the command "vxrecover -g -sb".

To rename diskgroups during deport/import, run the deport/import
command with the argument "-n ". After importing the diskgroup, run
the command "vxprint -hrt". The volumes that belong to the imported
disk group may be DISABLED. Run the command "vxinfo "volume name" to
be sure that the volumes are "startable". Then run "vxrecover -s" to
enable each volume. If the disks in question have been moved due to
the crash of the original system, then they will not have been
properly deported and will be locked. To unlock the disks, run the
command "vxdisk clearimport" or "vxdg -C import". Be sure not to run
these commands if the original system has physical access to the disks
since that would allow access to the disks from multiple hosts and
possibly cause data corruption.

How To: Mirror Removal
To remove a mirror from a volume (i.e., to remove one of the plexes
that belongs to the volume), run the following command:
vxplex -o rm dis

Any associated subdisks will then become available for other uses.
To remove the disk from Volume Manager control entirely, run the
following command:
vxdisk rm For example, "vxdisk rm c1t1d0s2".

How To: Volume Deletion
To remove a volume, unmount any filesystems on the volume and run
the following command: vxassist remove volume

How To: Mirror Backup
The following techniques can be used to backup mirrored volumes by
temporarily taking one of the mirrors offline and then reattaching the
mirror to the volume once the backup has been run.

1. Disassociate one of the mirrors from the volume to be backed up:
vxplex dis

2. Create a new, temporary volume using the disassociated plex:
vxmake -g -U gen vol tempvol plex=

3. Start the new volume:
vxvol start tempvol

4. Clean the new volume before mounting:
fsck -y /dev/vx/rdsk//tempvol

5. Mount the new volume and perform the backup

6. Unmount the new volume

7. Stop the new volume:
vxvol stop tempvol

8. Disassociate the plex from the new volume:
vxplex dis

9. Reattach the plex to the original volume:
vxplex att

10. Delete the temporary volume:
vxedit rm tempvol

To display the current Veritas configuration, use the following command:

To monitor the progress of tasks, use the following command:
vxtask -l list

To display information related to plexes, run the following
command: vxprint -lp

How To: Deport/Import
Disk groups can be moved from one system to another with the
deport/import commands. The same technique can be used to rename a
disk group by deporting the disk group with a new name.

1. Unmount any filesystems on the disk group

2. Stop any volumes on the disk group:
vxvol stop

3. Deport the disk group:
vxdg deport

4. Import the disk group:
vxdg import

5. Restart and resynchronize the disk group:
vxrecover -g -sb

6. Remount any filesystems from the disk group

If you have any problems, check the status of volumes on the disk group:

To rename the disk group during deport/import, use the -n argument:
vxdg deport -n


Soltice DiskSuite:

Solstice Disksuite is a free RAID volume manager from Sun Microsystems.
This article describes the method of mirroring a boot drive using
Solstice Disksuite via the command line.

Solstice Disksuite comes bundled with the Solaris 8 operating system.
Install Solstice Disksuite 4.2.1 on Solaris 8
Insert 2 of 2 software for Solaris 8

# CD /cdrom/sol_8_1000_sparc_2/Solaris_8/EA/products/DiskSuite_4.2.1/sparc/Packages

I usually install all the packages even though I do not use the GUI.

# pkgadd -d .

The following packages are available:
1 SUNWlvma Solaris Volume Management API's
(sparc) 1.0,REV=2001.
2 SUNWlvmg Solaris Volume Management Application
(sparc) 1.0,REV=2001.

3 SUNWlvmr Solaris Volume Management (root)
(sparc) 1.0,REV=2001.
4 SUNWmdg Solstice DiskSuite Tool
(sparc) 4.2.1,REV=1999.
5 SUNWmdja Solstice DiskSuite Japanese localization

(sparc) 4.2.1,REV=1999.
6 SUNWmdnr Solstice DiskSuite Log Daemon Configuration Files
(sparc) 4.2.1,REV=1999.
7 SUNWmdnu Solstice DiskSuite Log Daemon

(sparc) 4.2.1,REV=1999.
8 SUNWmdr Solstice DiskSuite Drivers
(sparc) 4.2.1,REV=1999.
9 SUNWmdu Solstice DiskSuite Commands
10 SUNWmdx Solstice DiskSuite Drivers(64-bit)
(sparc) 4.2.1,REV=1999.

Select package(s) you wish to process (or 'all' to process all
packages). (default: all) [?,??,q]: all

Reboot the server after the successful installation.

# init 6

Prepare the drives for SDS

SDS uses metadevice state databases to store information on disk
about the state of your DiskSuite configuration. The metadevice state
database records and tracks changes made to your configuration. These
databases must reside on a dedicated slice(in the case of a boot
drive). I typically leave about 10MB of unused space on the boot drive
when installing Solaris for these databases.

Disksuite has the following rules with respect to the use of
database replicas:

The system will not boot unless more than half of the replicas are
available The system will panic if more than half of the replicas are

If one of your drives fail, and the system is rebooted for any reason
it will not automatically boot in a two disk mirror configuration. You
can disable the feature by setting the following system parameter:

# echo "set md:mirrored_root_flag=1" >> /etc/system

The output from format of my boot disk looks like the following.
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 0 - 283 400.62MB
(284/0/0) 820476
1 var wm 284 - 1701 1.95GB (1418/0/0) 4096602
2 backup wm 0 - 24619 33.92GB (24620/0/0) 71127180
3 swap wu 1702 - 3136
1.98GB (1435/0/0) 4145715
4 unassigned wm 3137 - 3144 11.29MB (8/0/0) 23112
5 unassigned wm 3145 - 5271 2.93GB (2127/0/0) 6144903
6 usr wm 5272 - 11652
8.79GB (6381/0/0) 18434709
7 home wm 11653 - 24619 17.86GB (12967/0/0) 37461663

The partition table of the mirrored drive should be identical to
the boot drive. Copy the partition table of the boot drive to its

# prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2

fmthard: New volume table of contents now in place

In this case c0t0d0s2 is the boot drive and c0t1d0s2 is the mirror.

# metadb -a -f -c2 /dev/dsk/c0t0d0s4 /dev/dsk/c0t1d0s4

Create the mirror for / filesystem
# metainit -f d10 1 1 c0t0d0s0
d10: Concat/Stripe is setup
# metainit d20 1 1 c0t1d0s0
d20: Concat/Stripe is setup
# metainit d30 -m d10
d30: Mirror is setup
# metaroot d30

Create the mirror for all other filesystems
/var filesystem:
# metainit -f d11 1 1 c0t0d0s1
# metainit d21 1 1 c0t1d0s1
# metainit d31 -m d11

Swap filesystem:
# metainit -f d13 1 1 c0t0d0s3
# metainit d23 1 1 c0t1d0s3

# metainit d33 -m d13

/opt filesystem:
# metainit -f d15 1 1 c0t0d0s5
# metainit d25 1 1 c0t1d0s5
# metainit d35 -m d15

/usr filesystem:
# metainit -f d16 1 1 c0t0d0s6
# metainit d26 1 1 c0t1d0s6

# metainit d36 -m d16

/export/home filesystem:
# metainit -f d17 1 1 c0t0d0s7
# metainit d27 1 1 c0t1d0s7
# metainit d37 -m d17

Edit the /etc/vfstab to mount the new mirrors on boot.

Before :
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes -

fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c0t0d0s3 - - swap - no -
/dev/md/dsk/d30 /dev/md/rdsk/d30 / ufs 1 no -

/dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 /usr ufs 1
no -
/dev/dsk/c0t0d0s1 /dev/rdsk/c0t0d0s1 /var ufs 1
no -
/dev/dsk/c0t0d0s7 /dev/rdsk/c0t0d0s7 /export/home ufs
2 yes -

/dev/dsk/c0t0d0s5 /dev/rdsk/c0t0d0s5 /opt ufs 2
yes -
swap - /tmp tmpfs - yes -

After :
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes -
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/md/dsk/d33 - - swap - no -

/dev/md/dsk/d30 /dev/md/rdsk/d30 / ufs 1 no -
/dev/md/dsk/d36 /dev/md/rdsk/d36 /usr ufs 1 no -
/dev/md/dsk/d31 /dev/md/rdsk/d31 /var ufs 1 no -

/dev/md/dsk/d37 /dev/md/rdsk/d37 /export/home ufs 2
yes -
/dev/md/dsk/d35 /dev/md/rdsk/d35 /opt ufs 2 yes -
swap - /tmp tmpfs - yes -

Suppress warning messages
# metainit hsp001 Attach the second submirror to the mirror. This
will cause the data from the boot disk to be synchronized with the
mirrored drive.
# metattach d30 d20
# metattach d31 d21
# metattach d33 d23
# metattach d35 d25
# metattach d36 d26
# metattach d37 d27

you will hear alot of disk thrashing at this point and your i/o
will go through the roof.

Use metastat to track progress

# metastat I like to use this one when i'm watching though. it
looks more impressive
# metastat | grep % ...

Enable the mirror disk to be bootable:
# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0
# ls -l /dev/rdsk/c0t1d0s0
lrwxrwxrwx 1 root root 45 Sep 8 11:25
/dev/rdsk/c0t1d0s0 -> ../../devices/pci@1f,4000/scsi@3/sd@1,0:a,raw
This is the device path that you will use to define the alternate
boot device at the hardware level.
ok nvalias mirror /pci@1f,4000/scsi@3/sd@1,0:a,raw Issue a
"show-disks" at the ok prompt to verify the correct path to the disk.
Use "devalias" at the ok prompt to also give clues as to which device
path to use.

In case of primary boot disk failure, boot from the alternate disk

ok boot mirror

No comments: