Increasing NTFS Free Space Efficiency
Ξ February 22nd, 2007 | → | ∇ Uncategorized, Hardware, Windows, Tweaks |
Increasing NTFS Free Space Efficiency
A simple hard drive tweak for NTFS file systems.
Ok, this one is sort of old but not well known because many don’t understand what it does. Most folks should understand the difference between this tweak and what most would recognize as cluster size they see when formatting their hard drive in Windows XP/Vista.
Let’s go over something real quick.
When you format NTFS, you’re asked to input the allocation size. The default of which is 4096 bytes per cluster. This is known to be the most efficient for NTFS file systems which are expected to harbor both very small files and very large files alike. This is also the maximum cluster size that you can use that will support compression. As a side note, you’re able to format NTFS file systems with cluster sizes as large as 64 KB. The cluster size denotes how large of a swath of hard disk space a file will take up. If your cluster size is 64KB then any file under 64KB in size will take up 64KB on disk. For example, if you save a file that’s 11KB in size it will take up 64 KB on the disk. Most see this as a grotesque waste of space, however, this extreme cluster size is very useful on extremely large volumes that are designed to house very large files such as video files or databases. This also allows the Master File Table to be very small, helping you recover much of the space lost, also called MFT overhead, noticed by most users who format at the standard 4KB cluster size.
The tweak of which I speak refers nothing to this allocation size, but rather how the system handles free space.
WARNING: This involves editing the registry.. If you botch something up don’t blame me.
Who does this tweak apply to?
- Anyone using large hard drives or more specifically large volumes(100GB or bigger after MFT overhead, i.e. 120GB physical drive)
- Anyone who regularly stores, reads, moves, modifies or deletes large files. For a lot of you, this might include video files, uncompressed audio(for you studio heads out there.) and file archives including zips, rars, and any drive image backups you might store.
- Anyone with a lot of problems with fragmentation on their volumes due to lack of free space(more on that in a moment).
Who shouldn’t apply this tweak?
- Anyone with less than 100GB in a volume after MFT overhead.
- Anyone on Fat32 filesystem(there is a tweak LIKE this, but it’s buggy)
- Anyone looking to try this tweak on a Virtual hard disk such as those utilized by VMWare and MS Virtual PC. This can break the file that contains the virtual volume.
- Anyone who’s even slightly skittish about editing the registry.
The Tweak
What we’re going to is change the contiguous file allocation size. The default is something ridiculous like 16kB or something, meaning that when the system looks at the hard drive for free space to write a new file it comes across a gap of free space larger than 16kB it begins the new file there up to the end and then fragments to the next free 16kB. This can cause larger files to become insanely fragmented if stored on the same drive as many small files that are written and deleted often. What we want to do is set this value higher, a lot higher…
Depending upon what you do with the volumes, I’d initially recommend setting the value to 512kB. This means that now, the system will look for a contiguous length of at least 512kB of free space before beginning a file write operation. What does this do? A couple of things. Firstly, it speeds up the write process because the system will skip over the 16KB space and any other space less than 512KB. This also helps to avoid fragmentation because larger files will only fragment into spaces larger than 512KB.
On MY workstation, since I do a lot of video editing, but I also do a lot of writing and operations with small files, it behooves me to raise this value to 4096KB. This means that my system looks for a space of longer than 4 megabytes before writing a new file. This reduces fragmentation GREATLY and insanely speeds up the defragmentation process.
There are draw backs to this however. If your system can’t find the contiguous space long enough, then you may get a false “Drive is full” error. So this is risk of sorts. However, if you start this process while the drive has at least 50% free space and you keep up on defragging then you will have no problems.
Overall, this will increase the speed at which your system performs file operations on large files with very little, if any speed detriment to your small file operations.
Navigate your registry to the following key:
HKLM/System/CurrentControlSet/Control/Session Manager/Memory Management
Create a DWORD key named ContigFileAllocSize and give it the DECIMAL value of 512, 1024, 2048, or 4096 depending upon how large of a space you want to use for this process.
I do NOT recommend going above 4096. If you know what you’re doing, be my guest, but if you set this too high, you’re system may fail to boot.
After you create and value this key reboot your machine and defrag immediately with whatever tool you have handy. Preferably something that will defragment free space. You’ll notice the defragmentation process being super fast and efficient.
Contrary to popular believe, this isn’t not like setting up your cluster size to be bigger than what’s necessary. You do not LOSE free space with this tweak, but you may have the situation in which you will lose ACCESS to said free space due to heavy fragmentation.
Now… for you kids with super heavy defragementable volumes, this is because you no longer have contiguous space big enough to handle the movement of your files…
To remedy this set your contigfileallocsize to something absurdly low like 4 or 8. Reboot into safe mode and defrag… go read the Iliad, it’s going to take forever, but it will eventually finish. Once you’re all defragged, go back and set your key back up to 512 and defrag once more after rebooting. Also, don’t do anything, and I mean ANYTHING no web browsing, no email checking while you’re defragging at such a low contiguous allocation size. You’ll just seriously jack it up and cause MEGA fragments.
Remember, you have to reboot after changing this key value in order for it to take effect.
Good luck and God Speed.
on March 8th, 2007 at 4:54 pm
I have one simple question for this great piece of work.
Here you say
Create a DWORD key and give it the DECIMAL value of 512, 1024, 2048, or 4096 depending upon how large of a space you want to use for this process.
I created a DWORD key safely but what do you name the DWORD key you created?
on March 13th, 2007 at 8:39 pm
Question. Here you say,
Create a DWORD key and give it the DECIMAL value of 512, 1024, 2048, or 4096 depending upon how large of a space you want to use for this process. I do NOT recommend going above 4096. If you know what you’re doing, be my guest, but if you set this too high, you’re system may fail to boot.
What do you name the DWORD key anyways?
on March 14th, 2007 at 11:30 am
Sorry about that, somehow I managed to leave the key name out.
on March 20th, 2007 at 7:23 pm
How would this effect an F drive that was partitioned from the hard drive when the hard drive was formatted.
What I meant to say, does this just change how defragmenting works, since I have an F drive that I use more than my C drive but F is a shared hard drive for Linix. I know my F drive isn’t Windows only…