How to setup SVM using RAID 1 mirroring one disk to another

CONTENTS:

INTRODUCTION
PART I: DISK LAYOUT
PART II: METADB SETUP – FREEING SWAP SPACE FOR A NEW SLICE
PART III: VTOC COPY AND METADB INSTALLATION
PART IV: CONFIGURING OBP AND TESTING

——————————————————————————————

INTRODUCTION

This file is an attempt to explain how to setup RAID 1 on a Solaris 10 machine using
Sun Volume Manager. This was done on a Sun E4500 attatched by fiber to a StorEDGE A5100
array running Solaris 10 release 3/05, but should work on any Solaris system. I tried
to make this as easy to read as possible because all of the documentation I found was
hard to comprehend at first.

——————————————————————————————

PART I: DISK LAYOUT

The disks I will be using are c0t096d0 c0t097d0 for the Solaris OS and c0t098d0 c0t099d0
for the mirrors.

Here’s a layout of how I have c0t096d0:

0 root wm 293 – 514 389.26MB (222/0/0) 797202
1 usr wm 515 – 970 799.56MB (456/0/0) 1637496
2 backup wm 0 – 4923 8.43GB (4924/0/0) 17682084
3 var wm 971 – 1156 326.14MB (186/0/0) 667926
4 swap wu 0 – 256 450.63MB (257/0/0) 922887
5 unassigned wm 1157 – 2720 2.68GB (1564/0/0) 5616324
6 usr wm 2721 – 4923 3.77GB (2203/0/0) 7910973
7 unassigned wm 257 – 292 63.12MB (36/0/0) 129276

Here’s a layout of how I have c0t097d0:

0 unassigned wm 0 0 (0/0/0) 0
1 unassigned wm 0 0 (0/0/0) 0
2 backup wm 0 – 4923 8.43GB (4924/0/0) 17682084
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 home wm 0 – 4923 8.43GB (4924/0/0) 17682084

c0t096d0 is using all of the slices and c0t097d0 is filled completey with slice 7
(/export/home). This was automatically setup in the Solaris 10 install.

——————————————————————————————

PART II: METADB SETUP – FREEING SWAP SPACE FOR A NEW SLICE

If you already have a free slice and you don’t need to do this, skip down to PART II.

The first part in setting up a mirror is the metadb’s. These are databases placed on
slices that hold the information for the mirrors. It’s good to have these on their own
slice and its good to have atleast two of these on each disk. After installing Solaris
I realized I didnt have any free slices to use for the metadb’s so I had to shrink my
swap to allocate space for a new slice and then add swap back. This is how I did it:

First Boot into single user mode.

{5} ok boot -s

Once it’s booted, list our swap status:

# swap -l

swapfile dev swaplo blocks free
/dev/dsk/c0t96d0s4 118,76 16 1052144 1052144

This tells us it is on slice 4 (c0t96d0s4), we want to delete it:

# swap -d /dev/dsk/c0t96d0s4
/dev/dsk/c0t96d0s4 was dump device —
invoking dumpadm(1M) -d swap to select new dump device
dumpadm: no swap devices are available

List the swaps again to make sure it’s gone:

# swap -l
No swap devices configured

Enter the format utility (this is where you manage the disks):

# format

Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c0t96d0
/sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w210000203723015c,0
1. c0t97d0
/sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w21000020370ede66,0
2. c0t98d0
/sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w210000203714ef36,0
3. c0t99d0
/sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w21000020370bbfd9,0
4. c0t100d0
/sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w210000203716830d,0
5. c0t101d0
/sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w2100002037146891,0
6. c0t102d0
/sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w21000020370cf8dc,0
7. c0t112d0
/sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w2100002037139837,0
8. c0t113d0
/sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w210000203713bb84,0
9. c0t114d0
/sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w21000020371413ba,0
10. c0t115d0
/sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w2100002037141b20,0

Specify disk (enter its number): 0
selecting c0t96d0
[disk formatted]
Warning: Current Disk has mounted partitions.

partition> print
Current partition table (original):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 293 – 514 389.26MB (222/0/0) 797202
1 usr wm 515 – 970 799.56MB (456/0/0) 1637496
2 backup wm 0 – 4923 8.43GB (4924/0/0) 17682084
3 var wm 971 – 1156 326.14MB (186/0/0) 667926
4 swap wu 0 – 292 513.75MB (293/0/0) 1052163
5 unassigned wm 1157 – 2720 2.68GB (1564/0/0) 5616324
6 usr wm 2721 – 4923 3.77GB (2203/0/0) 7910973
7 unassigned wm 0 0 (0/0/0) 0

The swap partition is going from cylinders 0-292 so I want to fix that to be smaller.
Choose the swap to modify it.

partition> 4
Part Tag Flag Cylinders Size Blocks
4 swap wu 0 – 292 513.75MB (293/0/0) 1052163

Enter partition id tag[swap]:
Enter partition permission flags[wu]:
Enter new starting cyl[0]:
Enter partition size[1052163b, 293c, 292e, 513.75mb, 0.50gb]: 450mb
partition> print
Current partition table (unnamed):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 293 – 514 389.26MB (222/0/0) 797202
1 usr wm 515 – 970 799.56MB (456/0/0) 1637496
2 backup wm 0 – 4923 8.43GB (4924/0/0) 17682084
3 var wm 971 – 1156 326.14MB (186/0/0) 667926
4 swap wu 0 – 256 450.63MB (257/0/0) 922887
5 unassigned wm 1157 – 2720 2.68GB (1564/0/0) 5616324
6 usr wm 2721 – 4923 3.77GB (2203/0/0) 7910973
7 unassigned wm 0 0 (0/0/0) 0

This made it goto cylinder 0-256 instead of 0-292 so now I have 32 cylinders for the
unassigned slice we are about to create on number 7.

partition> 7
Part Tag Flag Cylinders Size Blocks
7 unassigned wm 0 512.00MB (292/0/0) 1048572

Enter partition id tag[unassigned]:
Enter partition permission flags[wm]:
Enter new starting cyl[257]:
Enter partition size[1048572b, 292c, 548e, 512.00mb, 0.50gb]: 292e
partition> print
Current partition table (unnamed):
Total disk cylinders available: 4924 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 293 – 514 389.26MB (222/0/0) 797202
1 usr wm 515 – 970 799.56MB (456/0/0) 1637496
2 backup wm 0 – 4923 8.43GB (4924/0/0) 17682084
3 var wm 971 – 1156 326.14MB (186/0/0) 667926
4 swap wu 0 – 256 450.63MB (257/0/0) 922887
5 unassigned wm 1157 – 2720 2.68GB (1564/0/0) 5616324
6 usr wm 2721 – 4923 3.77GB (2203/0/0) 7910973
7 unassigned wm 257 – 292 63.12MB (36/0/0) 129276

partition> label
Ready to label disk, continue? y

partition> quit

# reboot

After it reboots, check the slices again with format make sure they are the same.

——————————————————————————————

PART III – VTOC COPY AND METADB INSTALLATION

Now that we have the slices setup how we want them, we need to make them the exact
same on the mirror drive (c0t98d0). There is an easy command to do this:

# prtvtoc /dev/rdsk/c0t96d0s2 | fmthard -s – /dev/rdsk/c0t98d0s2
fmthard: New volume table of contents now in place.

After the slices are all setup in place on both disks, we are now ready to create
the metadb’s on both disks:

Remember above we allocated space from swap to use slice 7 for the metadbs so this is
where we will specify to use it. Here we are telling it to create two (-c 2) metadbs
on each disk.

# metadb -f -a -c 2 c0t96d0s7 c0t98d0s7

Create concat/stripe of slice 0 (c0t96d0s0).

# metainit -f d10 1 1 c0t96d0s0
d10: Concat/Stripe is setup

Create concat/stripe of slice 0 on mirror disk (c0t98d0s0).

# metainit -f d20 1 1 c0t98d0s0
d20: Concat/Stripe is setup

Create d0 and attatch submirror d10.

# metainit d0 -m d10
d0: Mirror is setup

For ONLY the root slice, you can use metaroot command to update the vfstab.
For other slices, you have to update vfstab by hand.

# metaroot d0

Reboot the machine. You have to reboot after running the metaroot command before
attatching the second submirror.

# reboot

Attatch the second sub-mirror (d20) to the volume (d0) which causes a mirror resync.

# metattach d0 d20
d0: submirror d20 is attached

To check progress of the attatchment:

# metastat | grep progress
Resync in progress: 12 % done

After this, the root slice is now setup to mirror and this can be done for all
other slices.
To double check your configuration type metastat and it should look similiar to this:

# metastat
d0: Mirror
Submirror 0: d10
State: Okay
Submirror 1: d20
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 797202 blocks (389 MB)

d10: Submirror of d0
State: Okay
Size: 797202 blocks (389 MB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0t96d0s0 0 No Okay Yes

d20: Submirror of d0
State: Okay
Size: 797202 blocks (389 MB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0t98d0s0 0 No Okay Yes

Device Relocation Information:
Device Reloc Device ID
c0t98d0 Yes id1,ssd@n200000203714ef36
c0t96d0 Yes id1,ssd@n200000203723015c

——————————————————————————————

PART IV – CONFIGURING OBP AND TESTING

After we have the root slice mirrored now, we can setup the OBP to boot from the mirror
root slice if the main one fails.

First get the device path for the mirror slice:
Write this down, we will need this for the OBP configuration.

# ls -al /dev/dsk/c0t98d0s0
lrwxrwxrwx 1 root root 74 May 11 16:05 /dev/dsk/c0t98d0s0 -> ../../devices/sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w
210000203714ef36,0:a

Reboot and go into the OBP.

# reboot

Now we’re in the OBP, we setup the alias called "fcalbackup" to point to our root mirror
slice that we wrote down.

{0} ok nvalias fcalbackup /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w210000203714ef36,0:a

Show the boot devices

{0} ok printenv boot-device
boot-device = disk diskbrd diskisp disksoc net

Now we set the boot device to boot from fcal then fcalbackup if it fails, then net.

{0} ok setenv boot-device fcal fcalbackup net
boot-device = fcal fcalbackup net

I also set the diag-device because my diags are always set to max.

{0} ok printenv diag-device
diag-device = fcal

{0} ok setenv diag-device fcal fcalbackup
diag-device = fcal fcalbackup

NOTE: If you mess up when you create a nvalias and have to delete a nvalias you must
reboot for it to show up when you issue devalias command.

If you want to test to see if it works, just do "boot fcalbackup" to boot from the mirror.

In the first guide part of the guide I explained how to just
mirror your root slice, and this is showing how to mirror the rest of the
slices.

So last time I left off with just mirroring the root slice (c0t96d0s0)
and now we will start with slice 1 (c0t96d0s1).

NOTE: In d21 2 is the submirror number and 1 is the slice number
same with d11, 1 is submirror #, 1 is slice #

Create concat/stripe of slice 1.

# metainit -f d11 1 1 c0t96d0s1

Create concat/stripe of slice 1 on the mirror disk.

# metainit d21 1 1 c0t98d0s1

Create d1 and attatch d11.

# metainit d1 -m d11

Edit your /etv/vfstab manually for other slices than root.
sample:

#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd – /dev/fd fd – no –
/proc – /proc proc – no –
/dev/dsk/c0t96d0s4 – – swap – no –
/dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no –
/dev/dsk/c0t96d0s6 /dev/rdsk/c0t96d0s6 /usr ufs 1 no –
/dev/dsk/c0t96d0s3 /dev/rdsk/c0t96d0s3 /var ufs 1 no –
/dev/dsk/c0t97d0s7 /dev/rdsk/c0t97d0s7 /export/home ufs 2 yes –
/dev/dsk/c0t96d0s5 /dev/rdsk/c0t96d0s5 /opt ufs 2 yes –
/dev/md/dsk/d1 /dev/md/rdsk/d1 /usr/openwin ufs 2 yes –
/devices – /devices devfs – no –
ctfs – /system/contract ctfs – no –
objfs – /system/object objfs – no –
swap – /tmp tmpfs – yes –

I changed the c0t96d0s1 entry to point to /dev/md/dsk/d1 and /dev/md/rdsk/d1 and thats it.

This tells SVM its OK to boot up even if only half of the metadbs are available:

# echo ‘set md_mirrored_root_flag=1’ >> /etc/system

# metainit -f d13 1 1 c0t96d0s3
d13: Concat/Stripe is setup
# metainit d23 1 1 c0t98d0s3
d23: Concat/Stripe is setup
# metainit d3 -m d13
d3: Mirror is setup
# metainit -f d14 1 1 c0t96d0s4
d14: Concat/Stripe is setup
# metainit d24 1 1 c0t98d0s4
d24: Concat/Stripe is setup
# metainit d4 -m d14
d4: Mirror is setup
# metainit -f d15 1 1 c0t96d0s5
d15: Concat/Stripe is setup
# metainit d25 1 1 c0t98d0s5
d25: Concat/Stripe is setup
# metainit d5 -m d15
d5: Mirror is setup
# metainit -f d16 1 1 c0t96d0s6
d16: Concat/Stripe is setup
# metainit d26 1 1 c0t98d0s6
d26: Concat/Stripe is setup
# metainit d6 -m d16
d6: Mirror is setup

Edit all other entries in /etc/vfstab like we did above.

#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd – /dev/fd fd – no –
/proc – /proc proc – no –
/dev/md/dsk/d4 – – swap – no –
/dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no –
/dev/md/dsk/d6 /dev/md/rdsk/d6 /usr ufs 1 no –
/dev/md/dsk/d3 /dev/md/rdsk/d3 /var ufs 1 no –
/dev/dsk/c0t97d0s7 /dev/rdsk/c0t97d0s7 /export/home ufs 2 yes –
/dev/md/dsk/d5 /dev/md/rdsk/d5 /opt ufs 2 yes –
/dev/md/dsk/d1 /dev/md/rdsk/d1 /usr/openwin ufs 2 yes –
/devices – /devices devfs – no –
ctfs – /system/contract ctfs – no –
objfs – /system/object objfs – no –
swap – /tmp tmpfs – yes –

# sync; sync; reboot

After it reboots, attatch the other submirrors to d1,d3,d4,d5,d6.

# metattach d1 d21
d1: submirror d21 is attached
# metattach d3 d23
d3: submirror d23 is attached
# metattach d4 d24
d4: submirror d24 is attached
# metattach d5 d25
d5: submirror d25 is attached
# metattach d6 d26
d6: submirror d26 is attached

# metastat | grep progress
Resync in progress: 0 % done
Resync in progress: 1 % done
Resync in progress: 16 % done
Resync in progress: 24 % done
Resync in progress: 18 % done

Leave a Reply

Your email address will not be published. Required fields are marked *