SUMMARY: Fixing misconfigured mirror

From: Steve Edberg <sbedberg_at_ucdavis.edu>
Date: Mon Jun 06 2005 - 09:18:47 EDT
To reiterate my problem:

I had a misconfigured mirror metadevice under Solaris 8/DiskSuite - 
the /opt filesystem was mounted on a raw partition making up the 
mirror (c1t0d0s5) instead of the metadevice (d5) itself. I had tried 
simply unmounting /opt and remounting it on the mirror device, but 
that resulted in a corrupted directory structure; doing a subsequent 
metadetach & metattach did not help.

My setup was: metadevice d5 was a mirror, consisting of two 
submirrors d15 (c1t0d0s5) and d25 (c1t0d1s5).

As it turns out, I was correct in the commands I needed but wrong 
about the sequence. What I did successfully was to (1) detach 
submirror d25, leaving only the single submirror d15 (which was the 
disk slice I had been using); (2) unmount /opt, remount onto the 
mirror d5; and (3) attach d25 back onto d5. After a few minutes of 
resyncing, all looks OK.

Now, I have to do this for /usr. My plan is to (1) edit vfstab so 
that /usr points to the mirror, (2) detach the second submirror, (3) 
reboot, (4) reattach the second submirror. After, of course, a 
verified backup. I also have to do this for swap.

Thanks to:

Ken Rossman
Wesley W. Garland
Peter Junker
Thomas M. Payerle
Alex Stade
Darren Dunham

In addition, I found this

	http://www.blacksheepnetworks.com/security/resources/labs/disksuite.html

- which summarizes disksuite usage well. And this -

	http://www.physics.umd.edu/rgroups/pnce/pcs-docs/Glue/disksuite/disksuite-boot-mirror.html

- which calls out some warnings about what I did wrong in the first 
place (not detaching mirrors before remounting onto the mirror).

	steve edberg


=======================
= Summary of replies: =
=======================

The thing to do here is to break the mirror (metadetach), then reattach it
(metattach).  You have to decide which side of the mirror is more correct
in this case, and leave that submirror attached to the mirror metadevice,
while detaching the other submirror.  But I'd make the assumption in this
case that it was the physical device that *was* mounted directly that you'd
consider correct in this case.
---------------------------------------------------------------
This should do the trick:

metareplace -e d5 c1t1d0s5

then "metastat | grep %" -- if it says "Resyncing, 0% done"  you're golden.

PS: Backup first!!
---------------------------------------------------------------
How to get issues with volumes under Solstice Diksmanager fixed lookup
up the documentation.

If you don't have it, the lookit up under http://docs.sun.com

Dealing with the file systems /, /usr, /opt and swap needs
special steps, otherwise there is going to be file system corruption.
---------------------------------------------------------------
I thinkyour bestbet is to get all problematic filesystems "unmirrored",
mounted using the physical device, then metadetach all submirrors but the
one corresponding to the physical device you were using, then metaclear
the mirror metadevice.

Then just "remirror"; make a submirror metadevice for the physical device in
use (this may be redundant, as probably still around) (if you do make 
it, use the -f option as it should be already mounted).  Create the 
mirror
metadevice with only a single submirror (the one corresponding to the mounted
physicsal device).  Now you must remount the filesystem so the mirror 
metadevice
is the mount point.  Then you can attach other submirrors.
---------------------------------------------------------------
I'll explain briefly what I believe has happened, and then tell you 
my conclusion.

When you first do a;

metainit -f d15 1 1 c1t0d0s5
metainit d5 -m d15

You're telling SDS/SVM that you've create a mirror and defined which 
drive to mirror _from_. When you then;

metainit d25 1 1 c1t1d0s5
metattach d5 d25

SDS/SVM knows which device to sync from. However, once sync is 
completed, things change. Unless you specified -g and/or -r when you 
created the mirror (from metainit(1M));

          The following read options for mirrors are supported:

          -g       Enables  the  geometric  read   option,   which
                   results  in  faster  performance  on sequential
                   reads.




          -r       Directs all reads to the first submirror.  This
                   should only be used when the devices comprising
                   the first submirror  are  substantially  faster
                   than those of the second mirror. This flag can-
                   not be used with the -g flag.

You'll end up with the default, which is a round-robin style read 
(from metainit(1M));

          If neither the -g nor -r flags are specified, reads  are
          made  in  a round-robin order from all submirrors in the
          mirror. This enables load balancing across  the  submir-
          rors.

So. My theory is that you're reading in a round-robin fashion, where 
the data on d25 is considerably older than the data on d15.

As for what you can do, I would (since you have a good backup) 
attempt to detach d25 from d5 and re-attach it;

metadetach d5 d25
metattach d5 d25

That should sync up the mirror from d15.
---------------------------------------------------------------
<SNIP>
>  My question is: is there an easy way to refresh a metadevice from a
>  physical slice so that I can simply remount onto the mirror device?

Yes.  metadetach the side that's not mounted, remount onto the mirror,
then metaattach the component back.

If you do this later, then corruption will be rewritten onto both sides
of the mirror.. *ouch*.

  I
>  know I could simply do a newfs on the d5 metadevice and restore the
>  backup, but I'd like to avoid that (especially since the /usr
>  partition is misconfigured this way as well). I assume pointing swap
>  at a metadevice will be no problem by editing /etc/vfstab & rebooting
>  will not be a problem since there's nothing persistent on swap, but
>  perhaps I'm missing something.

The most you could do at this point is to resync to one side (sounds
like you've done that) and fsck.  But that will only make everything
consistent, not necessarily "right".  The problem is that the corruption
has already occured, and probably been rewritten onto the disk....
</SNIP>


=====================
= Original message: =
=====================

Hi -

I have a Sparc/Solaris 8 system with some mirror metadevices set up 
using Solaris Disksuite. However, some of the filesystems were 
mounted via /etc/vfstab onto one of the physical disk slices that 
made up the mirror, rather than the mirror metadevice.

To remedy this, I first tested by unmounting the /opt filesystem 
(because my /opt was small, I had a current verified backup and there 
was nothing critical there), and remounting it onto the mirror 
device. In summary:

/opt originally mounted on /dev/dsk/c1t0d0s5
c1t0d0s5 one part of mirror d5; metastat -i d5 reports all ok
umount /opt
mount -F ufs /dev/md/dsk/d5 /opt

Now, I took a quick look at /opt and things *seemed* ok. However, 
problems soon became evident; eg, ls and ls -l reported different 
files/directories, trying to cd into some subdirectories of /opt gave 
me an IO error, recent files disappeared. I remounted /opt back onto 
the physical partition c1t0d0s5, and everything was OK again. It was 
as if the directory structure of the metadevice had not been updated, 
so anything (directories & files) that was older than approximately a 
year was visible. I am currently unable to trace things back to *why* 
the metadevice directory was updated then; it was over a year since 
the metadevices had been created, and much older than the last 
reboot; it *may* have been about the same time that the filesystems 
last needed fscking (thank <deity> for UFS logging!).

I had checked metastat -i, and mirror were reported all ok, so I 
assumed that things would be fine. I was wrong. I tried again doing a 
metadetach and metattach in hopes that would resync the mirror's 
directory, that didn't help either.

My question is: is there an easy way to refresh a metadevice from a 
physical slice so that I can simply remount onto the mirror device? I 
know I could simply do a newfs on the d5 metadevice and restore the 
backup, but I'd like to avoid that (especially since the /usr 
partition is misconfigured this way as well). I assume pointing swap 
at a metadevice will be no problem by editing /etc/vfstab & rebooting 
will not be a problem since there's nothing persistent on swap, but 
perhaps I'm missing something.

Thanks in advance; will summarize -

	steve edberg



Some details:

uname -a
========
SunOS pgfjupiter 5.8 Generic_117000-03 sun4u sparc SUNW,Sun-Fire-280R

Excerpt from current /etc/vfstab:
=================================
# device           device              mount         FS    fsck mount   mount
# to mount         to fsck             point         type  pass at boot options
#################  ##################  ############  #####  #   ### 
#############
/dev/dsk/c1t0d0s1  -                   -             swap   -   no      -
/dev/md/dsk/d0     /dev/md/rdsk/d0     /             ufs    1   no      logging
/dev/dsk/c1t0d0s6  /dev/rdsk/c1t0d0s6  /usr          ufs    1   no      logging
/dev/dsk/c1t0d0s5  /dev/rdsk/c1t0d0s5  /opt          ufs    2   yes     logging

metastat -p d5
==============
d5 -m d15 d25 1
d15 1 1 c1t0d0s5
d25 1 1 c1t1d0s5

(swap, /, /opt and /usr were all intended to be mounted on mirrors 
d1, d0, d5 & d6 respectively; only / is actually currently running on 
the mirror metadevice).

metastat -i
===========
d0: Mirror
     Submirror 0: d10
       State: Okay
     Submirror 1: d20
       State: Okay
     Pass: 1
     Read option: roundrobin (default)
     Write option: parallel (default)
     Size: 8389656 blocks

d10: Submirror of d0
     State: Okay
     Size: 8389656 blocks
     Stripe 0:
         Device              Start Block  Dbase State        Hot Spare
         c1t0d0s0                   0     No    Okay


d20: Submirror of d0
     State: Okay
     Size: 8389656 blocks
     Stripe 0:
         Device              Start Block  Dbase State        Hot Spare
         c1t1d0s0                   0     No    Okay


d1: Mirror
     Submirror 0: d11
       State: Okay
     Submirror 1: d21
       State: Okay
     Pass: 1
     Read option: roundrobin (default)
     Write option: parallel (default)
     Size: 1048707 blocks

d11: Submirror of d1
     State: Okay
     Size: 1048707 blocks
     Stripe 0:
         Device              Start Block  Dbase State        Hot Spare
         c1t0d0s1                   0     No    Okay


d21: Submirror of d1
     State: Okay
     Size: 1048707 blocks
     Stripe 0:
         Device              Start Block  Dbase State        Hot Spare
         c1t1d0s1                   0     No    Okay


d3: Mirror
     Submirror 0: d13
       State: Okay
     Submirror 1: d23
       State: Okay
     Pass: 1
     Read option: roundrobin (default)
     Write option: parallel (default)
     Size: 2889 blocks

d13: Submirror of d3
     State: Okay
     Size: 2889 blocks
     Stripe 0:
         Device              Start Block  Dbase State        Hot Spare
         c1t0d0s3                2889     Yes   Okay


d23: Submirror of d3
     State: Okay
     Size: 2889 blocks
     Stripe 0:
         Device              Start Block  Dbase State        Hot Spare
         c1t1d0s3                2889     Yes   Okay


d4: Mirror
     Submirror 0: d14
       State: Okay
     Submirror 1: d24
       State: Okay
     Pass: 1
     Read option: roundrobin (default)
     Write option: parallel (default)
     Size: 2889 blocks

d14: Submirror of d4
     State: Okay
     Size: 2889 blocks
     Stripe 0:
         Device              Start Block  Dbase State        Hot Spare
         c1t0d0s4                2889     Yes   Okay


d24: Submirror of d4
     State: Okay
     Size: 2889 blocks
     Stripe 0:
         Device              Start Block  Dbase State        Hot Spare
         c1t1d0s4                2889     Yes   Okay


d5: Mirror
     Submirror 0: d15
       State: Okay
     Submirror 1: d25
       State: Okay
     Pass: 1
     Read option: roundrobin (default)
     Write option: parallel (default)
     Size: 8389656 blocks

d15: Submirror of d5
     State: Okay
     Size: 8389656 blocks
     Stripe 0:
         Device              Start Block  Dbase State        Hot Spare
         c1t0d0s5                   0     No    Okay


d25: Submirror of d5
     State: Okay
     Size: 8389656 blocks
     Stripe 0:
         Device              Start Block  Dbase State        Hot Spare
         c1t1d0s5                   0     No    Okay

<SNIP>

-- 
+--------------- my people are the people of the dessert, ---------------+
| Steve Edberg                                http://pgfsun.ucdavis.edu/ |
| UC Davis Genome Center                            sbedberg@ucdavis.edu |
| Bioinformatics programming/database/sysadmin             (530)754-9127 |
+---------------- said t e lawrence, picking up his fork ----------------+
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers
Received on Mon Jun 6 09:23:32 2005

This archive was generated by hypermail 2.1.8 : Thu Mar 03 2016 - 06:43:47 EST