No, I’m not blogging about the upcoming Fraggle Rock: The Movie, which will probably be awesome. What I’m writing is WAY too geeky for Fraggles. If a Fraggle read this post, it would blow up. Fraggles are very Fragile.
If you use Microsoft Outlook in a corporate setting (ie, with an Exchange Server), and you have a LOT of email on the server, as in multiple gigabytes, AND you use Cached Exchange Mode, this post might be helpful to you. If you’re a geek and like to know how some things work under the surface, this post might be helpful to you. If you need help sleeping at night, and would like something to print out and read so you can actually get to sleep, this post might be helpful to you.
The plain truth is that although “defragging your hard drive” is way overrused as a troubleshooting step or performance tweak, there actually are times that it can be useful. In this post, I go into the more targeted use of defrag tools and when I’ve found defragging to actually be helpful.
If your computer is running slower than normal all of a sudden, and your local geek wannabe tells you to defrag your hard drive as one of the first things you do, you can tell him/her that that would just be wasting your time. There are times, however, like when you’ve filled up your hard drive all the way and also are working on big files, that may make defragging your drive in important step, but most slowdown problems now-adays are either due to spyware, viruses, or bugs. You can take steps to determine what’s eating up the performance on your Windows XP box, but it probably won’t be a defrag that fixes the problem.
Recently I’ve had a number of performance problems on my Windows XP machine, and most of the issues were Outlook-related. When Outlook was doing it’s thing – send/receive, sync, opening up email windows, etc – it basically locked up the whole machine for a few seconds. My eventual “fix” was to uninstall Office 2007 and re-install, with a service pack that fixes a key problem with large PST/OST files, but one of the steps I took along the way was to defrag my Outlook OST file. This can help tremendously when you A) don’t want to rebuild the OST from scratch over VPN and B) don’t want to defrag the entire drive.
More details are included in the extended entry, below. Read them Live free or die.
When Outlook is used in the optional Cached Exchange Mode, it gives many features for mobile users. Instead of Outlook being a real-time window into your server-based mailbox, it makes a copy of the mailbox on the local hard drive. In this mode, your interactions with Outlook have you dealing with the data on the local disk instead of the server, which may be very far away for a laptop user. It increases performance quite a bit, and also makes it possible for you to access all email, contacts, calendar, tasks when offline. In the background, Outlook quietly syncs up the mailbox on the server with the copy on the hard drive. This copy is called the “OST file” and it probably has a name like “outlook.ost”.
Let’s say you start off with a laptop with a very large hard drive. If you have a large mailbox on the server, you will have a large OST file on your laptop. When the hard drive is empty, this isn’t a big deal, because there’s plenty of room just about anywhere on the disk:
So let’s say that OST file finds a nice little home and becomes one of those thick blue bars. Later on, you add tons of files, do some movie editing, download a couple of albums, make backups, reload some things, and then after a while your hard drive is FILLED UP.
When the hard drive gets like this, and Outlook adds this email or that contact or this task into the OST file, the system tries really hard to keep that file as one big chunk on the disk, but because it’s putting stuff everywhere, it has to find somewhere else on the disk to put it. So it writes down a little marker so it knows where the next chunk starts, and puts data there. Then when it keeps adding data, it might find another instance where it simply can’t put any more in that place, so it writes down another little marker for where the next chunk starts, and it does this over and over.
Every time it writes down a marker and starts a new chunk, that is called a “fragment.” And everytime a fragment rings, a fraggle gets his wings.
This is important because the time it takes the hard drive to go from one spot on the disk to the next spot is very fast. However, the time it takes for it to go from one spot on the disk to some other spot somewhere else, is NOT very fast. It’s quite unfast, actually. So it makes sense that the more fragments you have, the longer it takes for the computer to actually read large files. When Outlook makes the hard drive head dance around just to read a stupid email, it makes a baby cry. When that email was just spam, it makes tons of babies cry.
You can defrag the entire disk at once, but that takes a while and might not fix your problem, especially if you don’t realize that if you have Outlook running, the system CAN’T defrag the OST file. It won’t touch a file that is in use. In fact, when you try, MC Hammer actually pops up in a window and starts singing “Can’t Touch This.”
There are two ways to have a contiguous OST file – delete it and rebuild it (which can take a long time if you’re on VPN) or defrag it.
Fortunately, Sysinternals has given us a utility, contig, that allows you to defrag a specific file. This is very useful when you don’t want the built-in defrag utility to move the file to the beginning of the hard drive – sometimes you just want it defragmented and to keep empty space after the file for growth.
Here’s a snapshot of me running it on my 3GB OST file, going from > 1400 fragments to 8.
C:\Documents and Settings\whall\Local Settings\Application Data\Microsoft\Outlook>contig -v outlook0.ost
Contig v1.54 - Makes files contiguous
Copyright (C) 1998-2007 Mark Russinovich
Sysinternals - www.sysinternals.com------------------------
Processing C:\Documents and Settings\whall\Local Settings\Application Data\Microsoft\Outlook\outlook0.ost:
Scanning file...
Scanning disk...
File is 814933 physical clusters in length.
File is in 1468 fragments.
Moving 160908 clusters at file offset cluster 0 to disk cluster 17715428
Moving 107070 clusters at file offset cluster 160908 to disk cluster 17219558
Moving 103878 clusters at file offset cluster 267978 to disk cluster 18162296
Moving 103522 clusters at file offset cluster 371856 to disk cluster 11159766
Moving 101563 clusters at file offset cluster 475378 to disk cluster 16871880
Moving 97102 clusters at file offset cluster 576941 to disk cluster 18635309
Moving 94042 clusters at file offset cluster 674043 to disk cluster 16974934
Moving 46848 clusters at file offset cluster 768085 to disk cluster 18912495
File size: 3337962496 bytes
Fragments before: 1468
Fragments after : 8
------------------------
Summary:
Number of files processed : 1
Number of files defragmented: 1
Average fragmentation before: 1468 frags/file
Average fragmentation after : 8 frags/file
If your OST file is split up into a large number of fragments, you will not be happy with Outlook, especially when you encounter a very serious if not-well-known bug in Outlook 2007 that makes disk accesses for large PST/OST files painfully slow and bloated. My life is so much better after reinstalling and loading the Service Pack that contained the fix.
If you actually read down this far, I might give you a hug.
You might want to give me a hug because I read the whole darn thing and while I may not have understood it ALL, I will refer to it if I am ever experiencing the problems you mentioned.
Okay, Wayne.
First off, know your audience. YOu are fantastic at giving technical presentations. I have had to learn the hard way how to know my audience.
Second, great post, for a techie. If you want some help on how to write for your audience, please let me know. you NEVER listen to the advice I give you. The only thing I ask of you this year is that you start listening to my advice.
Great post though!!!
PS: If your exchange server only has gigabytes (even a lot) on your server, good lord, how many people are using it? Everywhere I have worked, it’s been terabytes…
sourpuss, ***HUG***. Most people won’t experience these issues, but I blog about them and use categories and meta tags so that when the person who DOES encounter it can easily find my post.
Abs, first off, thanx for the compliments.
I’ll address the audience comment first, and then the exchange server storage comment.
I’m pretty sure we have different ideas on who my audience is, or for whom I was targeting this post. Let me know who you think my audience is. I agree to writing to an audience, and I believe you to have great thoughts and advice on it. But I don’t think that’s a problem I have, or at least this post is not an example of it. To help further define *my* intentions for this post, here are some points to consider: Not all of my blog entries are written for the “regulars.” Not all are intended to drive up traffic or get a comment conversation going. Not all are intended to be funny.
For this particular post, my targeted audience is the person out there who runs into outlook performance and has a large OST file. I sprinkle tags and phrases in the post to optimize it for the search engines, so when that poor soul googles it, they can find this post and hopefully get a few more steps to help. And maybe they’ll laugh while reading it (that’s important to me.)
Now, seeing as how MOST of my “regulars” couldn’t care less about Outlook performance or defragging it’s files, I take a couple extra steps to accommodate them. But that doesn’t mean I have to pick topics or always write for them.
Some of these steps include:
read themlive free or die”, “a fraggle gets his wings”, “it makes a baby cry”, and “MC Hammer actually pops up in a window.”) I figure even if someone gets tricked into reading about something they don’t care about, they can get a chuckle or two.I do appreciate you combining constructive criticism with compliments – you know how to be heard.
On the Exchange piece – I think you misunderstood me when I typed “[if] you have a LOT of email on the server, as in multiple gigabytes…” I meant the user, not the server. I have 3.5GB in my personal mailbox, and many >2GB PST files. This by any definition, is a lot for Outlook. MS just extended the maximum mailbox size to 20GB in Outlook 2003, and before that it was 2GB. Our servers have hundreds of GB’s.
Secondly, if you’ve worked with people with TERAbytes on a single Exchange server, then those people are embracing horrible Exchange practices. Absolutely horrible. For one, per MS’s recommendation (and my personal experience administering exchange servers for 7 years), a single exchange information store should never contain more than 30GB (some say 50GB). This is due to the amount of time users will be offline when the store gets corrupted. 30GB on today’s servers means 4-8hrs of repair or restore if it goes south (and all exchange information stores get corrupted at some point). Exchange Enterprise Servers can have four (4) information stores per Storage Group, and they can have four (4) Storage Groups, for a maximum of 16 information stores on a single server. For someone to have one terabyte on an exchange server, they’d have to average more than 60GB per information store, which is twice the recommended limit. If someone is doing that, then their users will be offline for 12+ hours when doing a repair on the store. Horrible practice if you ask me.
That’s why the answer is to have multiple exchange servers. We have eight (8) at our company, servicing 500 employees. We keep information stores around 20GB to allow for growth.
Believe me when I say you don’t want to have a 100GB information store that won’t mount, has 1018 errors, and your users are screaming for 36 hours while the fastest server in the company is trying to run a repair on the .edb file. That’s what I inherited back in 2003 when we merged with another company, and I quickly worked to distribute those to multiple stores on multiple servers.
You and Abs are funny…like brother & sister
thanks for making us “non-geeks” feel not so stupid
Christine:
You have probably nailed it right on the head. I never thought about it that way, but you know what? That’s EXACTLY how we are.
Except he thinks he is superior to me, because he is two years older than me, but I am superior to him. One day, he will realize this, or just give in out of pure exhaustion.
So, he is like that with everybody, good to know…lol
ya’ll crack me up!
Love ya Jer!
Thanks for the tip. Next question. Do you know if the file space is contiguous if your OST file is say… 2gb but only contains 1.5gb of files. If you defrag and do not compact now…does this keep the file from fragmenting during the next .5gb of data. I just cleared about .5gb of sync conflicts out and defragged and am wondering if compacting the file is actually a bad idea (like exchange..don’t reduce your edb space because having empty space in the storage database reduces file fragmentation)
Christine, yeah, we are very much like siblings – lots of fighting, a little bit of arguing, but a baseline of respect that comes by being family. You’re one of the very few people reading the blog who actually knows my REAL sibling, and he’s very different than Abs
and I still don’t know why your gravatar isn’t working – this is ridiculous! Gravatar themselves aren’t recognizing your gravatar, so if you want, I can help you make sure it’s set up. I’ll call you later to review.
Abs, my age is the LAST reason I’d use to justify my superiority
vcdechagn, first off, I need to know how to pronounce your name
. Secondly, yes, if you remove emails (content) from a .PST file but do not compact it, the file retains “empty space” within the file and new emails and content can take up that place. It actually will compact it by itself after a while during idle time. It’s the same way, as you mention, on Exchange with the .edb files. You can remove mailboxes and archive email all you want, but the .edb file will never actually shrink in size unless you bring it offline and compact it. This is another reason to keep the information stores small, as the compacting process is slowwww and is service-affecting.
Thanx for stopping by!
here’s where the name comes from. A girl I was dating in the younger days of the internet (early 90’s) used LttlLotte and VcdeChagn as our handles on IRC (old school SLIP and IRC on IRCII)
The characters are from Phantom of the Opera (Raoul, the Viscount de Chagny) and Christine (Little Lotte). Mostly I get called Vc for short.
Yeah, I’m leaving it the way it is. I have the space and outlook is SO much faster.
Thanks for calling to review my gravatar. I am sure I did something wrong. Thanks for trying to fix.
Wayne, you are a geek, but in a good way.
Are they really making a Fraggle Rock Movie? I am so there!
You better not be teasing!
vc, ah the old days of SLIP. Back on ‘The Lane,’ one of the “good” bbs’ in Austin, early 90’s, I was QG, which stood for “Quality Gophers,” which was an extremely obscure and nonsensical reference to a self-made-up phrase “Quality gophers in about an hour.” It means nothing, but I cracked myself up with it. Constantly. Boy, the kids these days don’t know how to waste time properly.
Christine, no problem. I believe I have an avatar ready for you -check your mail. In fact, I just tested it and it’s working now. Yay!
Raganator, I only know what I read on wiki and IMDB. The anticipated release is 2009.
Excellent blog entry. I had the exact problem (o.ost file 2.5GB with 1265 fragments). Running contig as I write this.
Thanks for the info.
Ed, glad to help! I must also say that moving to a new box (Vista SP1 with Office 2007 and no Google Desktop Search) has improved outlook performance DRAMATICALLY. I am completely in love with Vista and Office 2007 now.
Where is fine tool-recover outlook 2003 ost, this program is a reliable and efficient application, it will save many hours of your work and help your users to restore critical data,tool for Outlook recover .ost files, that can recover ost to pst file and save your time and money,will help you not to lose contacts of your partners, when Microsoft Exchange Server is out of order, all your message history will be also restored,can be used by anyone, from beginners to experts. Its intuitive graphic interface permit to perform all operations in several mouse clicks.
Thanks for the tip. This did exactly what it was supposed to and this article came up first in a google search. I just ran contig and the outlook.ost file went from 1,000 fragments to 2!
BTW, I had tried deleting and rebuilding the OST file just before this, but it rebuilt it with just as many fragments (and yes, I defragged the whole drive too after deleting it). I think that occasionally rebuilding that file is probably a good idea if you have the bandwidth, but it won’t necessarily fix the fragmentation.