SUMMARY Re: URGENT - /usr/lib/ missing

From: Steve Fick (
Date: Thu Mar 30 2000 - 13:50:56 CST

Thank you to everyone who replied.
Received several suggestions that will be very useful
in the future, both pro-actively and during emergencies.

Time-wise for this particular situation,
the suggestions arrive just a tiny bit
too late to improve on my proposed procedure,
which ended up taking 65 minutes end-to-end.
I had begun restoring the whole of /usr from tape to a disk
before I posted the original request. I received 5 responses
after the restored partition had been created but before
I had mounted it onto the troubled system. I decided
to go ahead and proceed with the original plan,
since it assured an up to date version, patch wise,
of the contents of /usr.

After I had gotten the troubled system running per
the original plan, I received another handful or so of responses
including a couple that could have reduced
the downtime to a lot less than 65 minutes,
if the only missing file had been /usr/lib/
(as it turned out, some additional files were absent today,
but I did not have a quick way to determine this
before I began the restoration process.)

Do not happen to have another E450,
nor another Solaris 2.6 box, nor a JumpStart server;
this disqualified some of the suggestions.

Special thanks to Casper Dik and Sam West
for telling me about /etc/lib/, I had been
unaware of it.

Below, the gist of the responder's comments.

==> Allan_West <==
>From Thu Mar 30 13:36:48 2000

>I accidentally removed that file once, and recovered quickly using
        boot cdrom
>mounting the disk (on /a) and then copying the file from the CD onto the

==> Brendan_Choi <==
>From Thu Mar 30 12:19:24 2000

>Did you run fsck -y ?

>If that don't work, just put in a Solaris CD-ROM (same version OS of
>course), boot off of that and find the /usr/lib/ file somewhere in
>the CD filesystem (I think it needs it too, right?), and also in the Solaris
>packages directory (its gotta be in there too right?).

==> Bruce_Wilkins <==
>From Thu Mar 30 11:21:41 2000

>Steve--you can boot from the cdrom; mount
>the file system and copy the library over
>from the CDROM.

>I know this works because it happend to me.

==> Casper_Dik <==
>From Thu Mar 30 12:32:14 2000

/usr/sbin/static/cp /etc/lib/ /usr/lib

==> Chris_Knox <==
>From Thu Mar 30 12:07:53 2000

>Your recovery approach sounds reasonable, however my larger recommendation
>would be to
>dispense with the nightly bounce. Whatever application that requires it is
>threatening the integrity
>of your system, as shown.

>If the bounce remains a requirement, I'd suggest mirroring the OS and also
>keeping a Jumpstart
>image around on another system and be ready to re-install over the network

==> David_Brierley <==
>From Thu Mar 30 13:04:24 2000

>If your primary concern is getting the machine up and running as fast
>as possible, boot the machine into single user mode and then NFS mount
>the /usr partition from another machine. You can then modify vfstab to
>use the NFS mount instead of the local disk and reboot the machine.
>You should be able to get the machine back up within 10 minutes. Once
>the machine is up you can use the tape backup to rebuild the local /usr
>and when you do your nightly reboot you can have it switch back to the
>original config.

==> David_Evans <==
>From Thu Mar 30 13:06:48 2000

>Rather than the whole of /usr just try coping /usr/lib/ or
/usr/lib. You will need to make sure the hardware is the same
>architecture and OS version.

==> Doug_Otto <==
>From Thu Mar 30 11:46:41 2000

>Begs the question why on earth would you have to reboot a Solaris box

==> Gerhard_den_Hollander <==
>From! Thu Mar 30 13:16:34 2000

>use a difefrent machine to tar to floppy
>boot the 450 from CD,
>untar from floppy.

>I've attached a and from our E450 (sol 2.6)

==> Haresh_Chandnani <==
>From Thu Mar 30 11:25:05 2000

>If you already have a same version of Solaris running on other system you
>will have to copy from that system. To do so you will have to boot the
>system with "boot cdrom" don't start the installation steps. Moment you get
>with GUI screen take your cursor outside the window right click mouse button
& click on Utilities -> Console option. Create a temp. directory in /a
>directory & mount the slice which has /usr/lib to temp directory created
>under /a change your working directory to /a/temp/usr/lib do ftp to the
>system having similar Solaris version & get this file.

==> Jonathan_Loh <==
>From Thu Mar 30 11:28:31 2000

>You could use the one from the cdrom. or you can boot into cdrom and
>assuming you have a net connection ftp it from another machine. Either way
>you may want to restore from backup afterwards.

==> Sam_Nelson <==
>From Thu Mar 30 13:49:48 2000

>My E250 (Solaris 7) has a file bit-for-bit identical to /usr/lib/ at
`/etc/lib/' and I didn't copy it there. Can you copy that to the
>place? Might be worth a try. On the other hand, if the /usr FS is broken
(and it sounds as if it may be) then maybe you won't be able to copy it
>You might be able to point LD_LIBRARY_PATH at /etc/lib long enoug to do a
>successful fsck of /usr? Can you mount your (possibly broken) /usr via
`mount -f' long enough to figure out what went wrong? You may need to boot
>from CD to try any of the above.

>For the future, I think you need to question the wisdom of rebooting
(unattended?) every night. It doesn't do your average Solaris box any good
>to be regularly perturbed like this...

==> Tim_Carlson <==
>From Thu Mar 30 12:15:29 2000

>If you have a jumpstart server running, do a

>boot net -- install

>and instead of going through the install, just open up a commandtool
>window and do the following

>mkdir /tmp/mnt
>mount /dev/dsk/c0t0d0s6 /tmp/mnt (guessing here on the slice number)
>cd /tmp/mnt/lib
>ftp someftpserver
>cd /usr/lib
>chmod 755

This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:14:05 CDT