Hey Everyone,
Do to the great response I needed to put out another summary including
the thanks to everyone, this is the 2nd and final summary to this
problem.
Thanks to all who responded:
Casper Dik <casper@holland.Sun.COM>
blymn@awadi.com.au (Brett Lymn)
rob.e.allan@hydro.on.ca (Rob Allan)
thenrion@csc.com (Timothy Henrion)
niziarj@itd.ssb.com (Richard J. Niziak)
David G Wiseman <magi@csd.uwo.ca>
Jens Fischer <jefi@kat.ina.de>
anderson@neon.mitre.org (Mark S. Anderson)
<DonWilliams@research.natpower.co.uk>
Cola Allen <cda@breacad.beckman.com>
Tom Schmidt <tschmidt@micron.com>
z055084@uprc.com (Kohler R. P. (Robert))
gary@ranchogrande.mce.com
ssa@pvdsw.amat.com (Ken Ferguson)
esilva@netcom.com (Eduardo E. Silva)
Glenn.Satchell@uniq.com.au (Glenn Satchell - Uniq Professional Services)
Darius Terrell/Avalon Software Inc <Darius_Terrell@khan.avalon.COM>
Scott Pleitner <scott@cs.curtin.edu.au>
Glenn Carver <glenn@atm.ch.cam.ac.uk>
nobroin@esoc.esa.de (Niall O Broin - Gray Wizard)
Here is my original message:
>Hey Everyone,
>
>I am having some trouble getting 14GB out of a tape using ufsdump with
>Exabyte's 8500XL 8mm tape drive, the 8500XL is supposed to be able to get
>14GB on a 160m tape but it stops at around 10GB. Here is the command I am
>using:
>
> ufsdump 0ubf 126 /dev/rmt/0c {partition}
>
>The system is a Sparc 20 running Solaris 2.4 with kernel patch 101945-34.
>
>Is there another device I should be using for more compression? I see
>in the Solaris 2.5 "st" man page is has a "u" device that is says is
>"ultra/compressed" but it doesn't say if it is compatible with the
>8500's. Is this also the case for Solaris 2.4?
>
>All/Any input would be greatly appreciated.
>
>Thanks in advance for all your responses.
>
>Later,
>
>Mark
>
>===========================================================================
>Mark Sturtevant <sturt@winternet.com> <http://www.winternet.com/~sturt>
>Unix Systems Administrator
>P A R A N E T
>===========================================================================
The General concensious is that 10-11GB on a 160M tape using the 8500XL is
normal, the 14GB is a virtual number that the company came up with using
data that probably wasn't compressed and is based on a 2 time compression ratio,
the real world ratio (with a mixture of data types) that most of the people
wrote me back with is 1.6, which gives you around 10GB.
The following are the responceses I received:
-----------------------------------------------------------------------------
From: Casper Dik <casper@holland.Sun.COM>
On my systenm, the u devices are the same devices as the c devices.
Casper
-----------------------------------------------------------------------------
From: blymn@awadi.com.au (Brett Lymn)
It really depends on the data you are backing up. The native
(i.e. uncompressed) capacity of your tape drive with 160M tapes is
7Gig. The 14Gig comes from exabyte (and others) assuming a 2:1
compression ratio. In my experience, with a normal mix of text and
binary files, you should expect about 1.5:1 for a compression ration
which is about what you are getting on your tapes.
-- Brett Lymn, Computer Systems Administrator, AWA Defence Industries =============================================================================== "Upgrading your memory gives you MORE RAM!" - ad in MacWAREHOUSE catalogue.----------------------------------------------------------------------------- From: rob.e.allan@hydro.on.ca (Rob Allan) X-Status:
The actual capacity of this type of tape is 7Gb, with 2X compression, you'll get 14 Gb. It is very rare to get this level of compression if you are backing up a file system which contains binaries, already compressed files, etc.
---------- Rob Allan | rob.e.allan@hydro.on.ca Ontario Hydro | Tel. (416) 592 4195 Toronto, Ontario, Canada | Fax (416) 592 4966 ----------------------------------------------------------------------------- From: thenrion@csc.com (Timothy Henrion)
Mark,
I've found Exabyte's compression claims to be a little exaggerated. My guess is that you won't get more on a tape than you are already getting.
-Tim
-- Tim Henrion Computer Sciences Corporation thenrion@csc.com ----------------------------------------------------------------------------- From: niziarj@itd.ssb.com (Richard J. Niziak)
The 14 Gb capacity is their best case scenario, unless you are backing up raw databases or just text files, I doubt that the compression algorithm that they use will get you more than 10-11Gb on the tape..
BTW the "u" or "ultra" device is only for the 4mm dat drives...
-Rick Niziak Systems Consultant Hammerhead Consulting ----------------------------------------------------------------------------- From: David G Wiseman <magi@csd.uwo.ca>
Five'll get you ten that the data you were trying to back up was already compressed somewhat... I've already encountered this on all of my compressed backup drives: we do a lot of image/vision work and those files are already so compressed that the drive s/w cannot compress them further. It got so bad here that I don't even try to compress any more.
-- magi David Wiseman, Network Manager e-mail: magi@csd.uwo.ca Department of Computer Science The University of Western Ontario fax: +1 519 661 3515 London Ontario Canada N6A 5B7 voice: +1 519 679 2111 x6879 ------------------------------------------------------------------------------- Supporting Windows is like buying a puppy. The dog only cost $100, but we spent another $500 cleaning the carpet. -- Marc Dodge ----------------------------------------------------------------------------- From: Jens Fischer <jefi@kat.ina.de>
14 GB is an estimated value which relies on a compressin factor of 2. As compression factors are varying depending on the data which is compressed the 14 GB capacity is not always true. You may reach this capacity when you are storing large ascii files. In our site we are experiencing a compression factor of about 1.5, which leads to an amount of 10.5 GB per tape. You should also be aware of the space which is produced between two logical volumes on one physical tape, which needs an additional amount of some MB each.
Hope that helps
Regards - Jens Fischer
I User : Jens Fischer Department : DV-Anwendungsentwicklung Technik I N A Company : INA Werk Schaeffler KG Address Industriestrasse 1-3 A D 91074 - Herzogenaurach Phone : (+49)9132-823262 FAX : (+49)9132-824953 e-mail : fischjns@kat.ina.de
----------------------------------------------------------------------------- From: anderson@neon.mitre.org (Mark S. Anderson)
First of all, you will not get 14GB. Due to overhead, you will only be able to use approximately 90% of the tape for data. So, you're down to 12.6GB max. Then, as you probably know, the 14GB figure ASSUMES a 2:1 compression ratio. You may or may not achieve this. If you are getting 10MB on the tape, and assuming a 10% loss as mentioned above, then the compression ratio is about 1.6. This is not unreasonable.
How I estimated the compression ratio: You have a tape that will hold 7GB without compression. If 10GB is 90% of the tape, then the tape is holding 11.11GB. The compression ratio is 11.11/7 ~= 1.6
Mark Anderson ---------------------------------------------------------- Mitretek Systems, Inc. manderso@mitretek.org 7525 Colshire Drive, MS Z420 voice: (703) 610-1762 McLean, VA 22102 FAX: (703) 610-1561 ----------------------------------------------------------------------------- From: <DonWilliams@research.natpower.co.uk>
Don't ever believe any supplier's claims about how much data fits on a tape using compression. When you go back to them they'll simply say "well, it depends on your data". Some data is more compressible than other data. We have a large hetrogenius network with what I consider to be fairly 'normal' data, and we have never ever managed to get the advertised capacity onto any compressed tape device from any supplier. It makes me wonder what kind of data they use in their tests to make these claims. ----------------------------------------------------------------------------- From: Cola Allen <cda@breacad.beckman.com>
Mark,
The type of data you are writing to tape determines the amount of data that can be stored. ASCII data will compress significantly whereas graphic files like GIF and binary files and obviously compressed files will not compress much whether compressed by h
ardware or software. I believe the 14GB figure is an estimate based on some reasonable mix of data. 10GB doesn't sound to far out depending on your data.
Hope this helps.
Best regards, ^ ^^ ^ ^^ ^^ ^ ^ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+ | ||^ | || ||^| | |e-mail cda@breacad.beckman.com| |^||| | | ||||^| |phone (714)961-6553 | ||||| __o |||||| +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+ . .~_( <, .~. .~. . _ (*)>(*) _ _ _ _ Cola Allen . . . . . . . . .
----------------------------------------------------------------------------- From: Tom Schmidt <tschmidt@micron.com>
Mark,
The 14GB capacity is assuming a 2:1 compression ratio. If your filesystem contains a lot of files that are not highly compressable (.zip, .gz, .Z, .gif, .jpg, etc.) then you may not achieve this ratio of 2:1. 7GB is the uncompressed capacity. On my 8500XL, I typically get about 11GB on a 160m tape. Your mileage may vary. :)
I have not tried the "u" option on the tape drive with my 8500XL. I just use the "c" option.
Tom
-- _____ ___ Tom L. Schmidt, Manager, Component Characterization | | / \ Micron Technology, Inc. | | \___ 8000 S. Federal Way P.O. Box 6 Mail Stop 376 | | \ Boise, Idaho USA 83707-0006 | |____\___/ tschmidt@micron.com
----------------------------------------------------------------------------- From: z055084@uprc.com (Kohler R. P. (Robert))
I've never heard of an 8500XL drive.
I've heard of 8505XL and 8500C...
8505XL's are half height drives with 7GB native/14GB compressed ratios.
8500C's are full height drives with 5GB native/10GB compressed ratios.
Are you sure you don't have an 8500C??? (Easy way to tell if it is a full height or a half height drive...)
If you indeed have an 8500XL drive, I'd be interested where you got it, because I have a robotics system that requires full height drives, thus I can't upgrade it to the 8505XL drives. But 8500XL sounds nice if it exists..
Anyway, I have about 25 - 8505XL drives on about 12 SUN/Solaris 2.4 systems.
You must make the kernel mod in /kernel/drv/st.conf file
....
tape-config-list = "EXABYTE\040EXB\0558505","EXB-8505 from Exabyte","EXB_8505" ; #EXB_8505 = 1,0x35,1024,0x1679,4,0x14,0x15,0x8c,0x8c,3; EXB_8505 = 1,0x35,0,0x1679,4,0x14,0x15,0x8c,0x8c,3;
....
Note, I had to use 0 block length for compatibility purposes of Solaris 2.3 backup tapes.
I've successfully backed up 13 GB to a tape (granted the machines are all large Database Servers, thus can compress at 2:1 ratios).
I use the /dev/rmt/0hbn devices..
If you are using the 8505XL and everything else above is correct, I'd say that your data is already compressed somewhat, therefore you can only get say 1.5:1 compression ratio of the drive.
Let me know about the 8500XL!!!
later
_______________________________________________________ /\-==-==-==-==-==-==-==-==-==-==-==-==-==-==-==-==-==-=-\ L_@~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~@ | Robert Kohler | | Union Pacific Resources | | P.O. Box 7, MS3503 | | Fort Worth, TX. 76101-0007 | | | | OFFICE VOICE: (817)-877-6993 | | FAX: (817)-877-6598 | | WORK(7-4) E-MAIL: robert.kohler@uprc.com | __@~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~@ \/______________________________________________________/
----------------------------------------------------------------------------- From: gary@ranchogrande.mce.com
Are you using a 160m "XL" tape with the drive? There should be a "RS" logo somewhere on the tape.
2:1 compression should put about 10GB on a 112m tape and 14GB on a 160m Tape.
________________________________________________________________________ Gary W. Cook Director of Technical Services Minicomputer Exchange, Inc. 610 North Pastoria Avenue Sunnyvale, CA 94086 Email: gary@mce.com Tel: 408-733-4400 Fax: 408-733-8009 -----------------------------------------------------------------------------
From: ssa@pvdsw.amat.com (Ken Ferguson)
I would hope you mean an 8505XL ( half height unit not full height). I've always used 0hn for high density no rewind. I would expect that you should get about 12gb out of the tape depending on what you are writing. 14gb is a best case example. Try the 0h and see if that helps at all.
Ken ___________________________________________________________________________ | | | Kenneth Ferguson Applied Materials, Inc. | | Unix Systems Administrator PVDIS&T | | Tel: 408-563-3141 4250 Burton Drive | | Fax: 408-563-3598 M/S 2450 | | Email: ken@pvdsw.amat.com Santa Clara, CA 95054 | |___________________________________________________________________________|
----------------------------------------------------------------------------- From: esilva@netcom.com (Eduardo E. Silva)
Mark, compression is kind of trivial. You might get 2:1 or ?:1 ratios, depending on the types of data you are sending to tape. Some ofiles I have sent to tape had a 20:1 ratio! others less than 30%. Most ASCII, and DB tables with lots of holes will have a 3:1 ratio. Most binaries 70% ( not even 2:1).
Also if your are backing up ALREADY compressed files. They will get larger on tape. Also if you are sending lots of litle files, your tape capacity will decrease because of the intergap between files.
Hopes that helped!
Thanks!
-Ed _ /\o/\ / <_> \ /^^/ \^^\ /___\ +---------------------------------------------------------------------+ | Can you see them all around us? | +---------------------------------------------------------------------+ | esilva@netcom.com | +---------------------------------------------------------------------+ ----------------------------------------------------------------------------- From: Glenn.Satchell@uniq.com.au (Glenn Satchell - Uniq Professional Services)
The actual volume you get on the tape depends on how well your data compresses. So, the capacity will vary depending jon whether you are doing mainly text or binary files, for example. I think the marketing people specify a slightly optimistic number too.
regards, -- Glenn Satchell glenn@uniq.com.au | There's a fine line Uniq Professional Services Pty Ltd ACN 056 279 335 | between fishing and PO Box 70, Paddington, NSW 2021, (Sydney) Australia | standing on the shore Phone 02 380 6360 Pager 016 287 000 Fax 02 380 6416 | looking like an idiot.
----------------------------------------------------------------------------- From: Darius Terrell/Avalon Software Inc <Darius_Terrell@khan.avalon.COM>
Mark,
Try using the high density device, "/dev/rmt/0h"
----------------------------------------------------------------------------- From: Scott Pleitner <scott@cs.curtin.edu.au>
Exactly how much you get on the tape is dependent upon the type of data. The 14G is at a compression of 2:1 which is very unlikely. We have two exabyte drives and normally get about 1.6:1 compression. A quick bit of maths shows that you are experiencing about the same. We are running our exabytes of a SunOS box, so I can't help with the device question...
s. -- Scott Pleitner Systems Programmer Voice: +619 351 7667 Department of Computer Science Fax: +619 351 2819 Curtin University of Technology Email: scott@cs.curtin.edu.au Perth, Australia ----------------------------------------------------------------------------- From: Glenn Carver <glenn@atm.ch.cam.ac.uk>
Hi Mark,
Easy one this. I put out the same message in Dec 1995. Here's the SUMMARY. Basically, if you are getting 10GB you are doing well. The success of the compression depends very heavily on the type of files you are compressing. I only get 9Gb out of mine!
Cheers, Glenn
_______________________________________________________________________ Dr. Glenn Carver, Phone: +44 (1223) 336524 Centre for Atmospheric Science, Fax: +44 (1223) 336473 Cambridge University, Chemistry Dept., glenn@atm.ch.cam.ac.uk Lensfield Road, Cambridge, CB4 4JZ, UK http://www.atm.ch.cam.ac.uk/~glenn/
"Time is a great teacher but unfortunately it kills all its pupils" Hector Berlioz (1803-69) - French composer. _______________________________________________________________________
----- Begin Included Message -----
>From glenn Tue Dec 12 10:34:15 1995 To: sun-managers@eecs.nwu.edu Subject: SUMMARY: 14Gb Exabyte trouble with Copilot Cc: Andy_Feldt@phyast.nhn.uoknor.edu, jd@lri.stjosephs.london.on.ca, jwa@nbs.nau.edu
Apologies for the late summary. I've been caught up in end of term reports. I had few responses to this message. Andy Feldt wrote saying he gets 20% extra with compression on at best. J. Davies wrote and said he only gets about 10Gb on a tape.
In my case, I only managed to get 9.1GBytes on a XL tape (7GB uncompressed). That's 30%, so I was doing better than Andy! Upon further investigation however, I found that my concientious users have already compressed large numbers of files, particularly in 3 (each 2.1GB) of the filesystems that were being dumped.
The conclusion seems to be that, like a Mflop rating the capacity of compression Exabyte drives is a manufacturers figure than it is guaranteed you'll never get. In practise, it will be much less than this. Figures of 20-40% extra above uncompressed dumps have been reported.
Thanks to: Andy_Feldt@phyast.nhn.uoknor.edu (Andy Feldt) jd@lri.stjosephs.london.on.ca (J. Davis) jwa@nbs.nau.edu (James W. Abendschan)
------------------------------------------------------------------------ Dr Glenn Carver Email: glenn@atm.ch.cam.ac.uk Centre for Atmospheric Science Phone: (44-223) 336524 Chemistry Department Fax : (44-223) 336473 Cambridge University Web : Cambridge, Lensfield Road, http://www.atm.ch.cam.ac.uk/~glenn CB2 1EW, UK ------------------------------------------------------------------------
Original message:
I've got a problem with getting the 14Gbyte capacity of the new(ish) XL Exabyte drives when I use Backup Copilot to dump filesystems. Yet as far as I can tell, everything's right.
My last dump only gave me 8Gb before it reached the end of the tape. I've used Dan Schales (excellent) web page (http://aurora.latech.edu/sunman.html) to search through previous summaries. These seem to suggest that compression gives a 5:3 ratio in practice rather than the 2:1 the manual claims. But this should still give me about 11/12 Gbytes.
The system is a SPARCSserver 10/61 running 4.1.3.
Things I've done:
1. I've patched the kernel as described in the user guide for the 14Gb device to turn on the compression. I'm using /dev/nrst20.
2. I've checked compression is really on - I get the amber light on the device.
3. I'm using pukka Exabyte 160m XL tapes (brand new).
Here's the command Copilot uses (Copilot's own dump command):
/usr/etc/dump boLVpUuI0Mf 112 /tmp/lfile.Caesar.10371 1 Caesar /etc/dumpdates.Caesar_f glenn /dev/nrst20 /
4. You'll note the blocking factor, 112, is correct for the Exabyte.
5. There's no size option set on dump so the tape will run to the physical end of media.
Frankly I can't figure out what's going on. Most of the filessystems are comprised of text files so I should be getting much more than 8Gb.
If anyone else is using these drives with Copilot I'd be interested to hear from them. I will summarise.
Thanks, Glenn
----- End Included Message ----- -----------------------------------------------------------------------------
From: nobroin@esoc.esa.de (Niall O Broin - Gray Wizard)
This is a FAQ but it's not in the FAQ list, though it should be ! You're using the highest compression minor device, your problem is that your data is not as compressible as the marketing people would like it to be :-) In the real world, people seem to get a compression rate of between 1.3 and 1.6 to 1. Data such as JPEG files, compressed archives are basically uncompressible whereas sparse databases can be very compressible - your mileage will of course vary.
Kindest regards,
Niall O Broin
UNIX Network Administrator, Stations and Communications Engineering Department European Space Operations Centre nobroin@esoc.esa.de Darmstadt, Germany Ph./Fax +49 6151 90 3619/2179
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:10:55 CDT