Skip navigation

Introduction (2017)

I’m copying this advice website text here because I’m canceling its domain name (copyright-infringement-notice.com). This was written in 2012 and last updated in 2013. The advice given may be outdated or inaccurate. Links to on-page anchors and some unnecessary text were removed. I’m preserving it for historical value so that 20 years from now we can look back on it and remember the ridiculous era of copyright trolls and settlement scams more accurately. Consider it an entertainment piece, if you will.

I am not a lawyer so remember that everything you read here could easily just be me talking out of my ass about something I know nothing about! 🙂


How To Handle A Copyright Infringement Notice

If you are freaking out or desperate, skip directly to damage control. Read the rest after you’re done! Also look over the list of companies who engage in copyright-related threats, lawsuits, or hire others to do so on their behalf.

Copyright settlement letters and offers should generally be ignored; despite the threat of a lawsuit, a very small minority of offer recipients actually get sued. THE NAME OF THE GAME: MAINTAIN SILENCE. If they can’t confirm that you ever received the notice, they’re even less likely to go after you.

If you receive a copyright infringement notice, don’t respond to it, don’t visit the website provided, ignore threats of lawsuits and settlement offers, and if you are actually distributing the material in question, STOP IMMEDIATELY.

The best way to avoid such a notice is prevention and education. Learn what triggers these notices and how to prevent them. If you have a wireless router and it is not secured or password-protected, you need to lock it down immediately. If you run peer-to-peer file sharing software, it is a great idea to check for material being offered for upload by the software and remove it. Also, if the ports being used by the software are reconfigurable, change them so that your active software no longer uses the port number in the notice.

Note that ISPs are required to forward DMCA notices to you to maintain “safe harbor” status. Charter Communications, in particular, is notorious for mailing postcards claiming to be from “Charter High-Speed Internet Security Team,” requesting that you visit security.charter.com and punch in a message ID and your Charter account number. These are DMCA notices, but look like they have something to do with identity theft or viruses or spam relaying being detected.

NEVER RESPOND TO COPYRIGHT SETTLEMENT OFFERS OR COPYRIGHT INFRINGEMENT NOTICES. THIS INCLUDES VISITING A WEBSITE AND/OR ENTERING A PROVIDED CODE. BY ENTERING THE CODE, THEY CAN CONFIRM THAT YOU RECEIVED THE MESSAGE!

“I already gave them my name/entered the code/called them/visited their site! What do I do?!” Relax! You still have factors working in your favor that make it very unlikely you’ll actually be sued. Remember that they’re in this game to make money, and you’re supposed to “chicken out” and cough up some money without them ever having to prove their case in court. If you don’t pay up, the copyright troll will have to decide between letting it slide and focusing on easier prey, or paying a few hundred dollars in filing fees plus the daily salary for a real lawyer to stand in that courtroom. A settlement yields huge profits, while a court battle can actually end badly for the trolls who may be “on the hook” for your legal fees if they lose.

Aside from the risks and costs involved in going after you officially, consider the proof that they would need to produce. Your confirmed receipt of a postcard or email with a code and entering the code does not in any way constitute admission of guilt. Essentially, if you don’t do something that is tantamount to saying “I am guilty and my name is Samuel Manchester Oakenfold IV in Farmville, VA; send the bill to this address…” then you haven’t contributed enough to prove that you infringed. Knowing the name behind the IP address is the first step in a lawsuit (since defendants must be named to be sued), but simply knowing the name of the IP address user doesn’t make you guilty either.

The burden of proof in civil cases is lower than in criminal cases. Civil requires “clear and convincing proof” while criminal requires “proof beyond a reasonable doubt.” The suing party must show “clear and convincing” evidence that you infringed on the copyright of their client(s) to win the case. If you don’t admit guilt, don’t have infringing materials on your hard drives (which will be subpoenaed), and don’t give out any information otherwise such as in a forum or on a blog, the only evidence that exists is a shaky claim of the IP address you were assigned having participated in infringement. So far, everyone who has lost in an online copyright infringement case (i.e. Jammie Thomas-Rasset) has in some way or another admitted guilt and then attempted to frantically backtrack; no case exists that has been won solely on the merits of an IP address being known and alleged to have been used for infringement.

“I was a complete idiot and said or wrote something that I think could be considered an admission of guilt! Help!” Don’t panic. The first thing to do is to firmly take a position of silence. If you have admitted guilt or said something that could be construed as an admission of guilt, you are at a higher risk of litigation (though you still have the costs and risks working in your favor, those risks are lower for the copyright troll since they may now have enough proof to convince a jury that you did it.) GET A LAWYER. I’M SOME PERSON ON THE INTERNET, NOT A LAWYER. Now, if you’re going to be a complete moron and not at least consult with a professional, there are some things you can do to lower your risk, particularly in the event that things actually go to trial.


THE QUICK AND DIRTY GUIDE TO COPYRIGHT INFRINGEMENT DAMAGE CONTROL

The suggestions in this guide, if actually followed, shall be performed entirely at your own risk. This guide IS NOT TO BE CONSIDERED LEGAL ADVICE and you should GET A LAWYER! It’s possible that some of the actions suggested aren’t actually a good idea. By reading it, you certify that you understand that this guide is not intended as legal advice and agree to indemnify and hold the author harmless from any action stemming, directly or indirectly, from reading this guide.

So you’re freaking out and feel a desperate need to do some damage control? This short guide will help you understand different facets of the problem you face, and what you could possibly do to minimize risk. In a panic, some people will do things that actually make things worse, so TAKE A DEEP BREATH BEFORE READING FURTHER. When you’re calmed down and ready to tackle the task of damage control, you may proceed. We’ll cover this point by point, and we’re going to assume you are probably going to court so that you’re fully prepared if you actually end up there. Ready?

KEEP EVERYTHING YOU RECEIVE: Postcards, electronic mail messages, and any other communication between yourself and any other related parties needs to be archived somewhere safe for your own reference and/or to present during the court case (if it even makes it that far!) Destroying letters or deleting emails or failing to record telephone calls ONLY HURTS YOUR ABILITY TO DEFEND YOURSELF. You cannot “make it go away” by destroying the letter or message and pretending it was never received. You can just as easily print a message, stuff it in a bank safe deposit box, and then proceed to enjoy your right to keep your mouth shut and pretend like you never received the letter if it is in your interests to do so. For all you know at this stage, the letter you might be afraid of and inclined to throw away may contain information that proves you’re innocent!

REALLY DELETE INFRINGING MATERIAL: Unlike communication between you and others, any potentially infringing data on your computer is almost certainly NOT going to prove that you aren’t guilty of copyright infringement. The problem is that it’s nearly impossible to RELIABLY destroy traces of the files that you delete. When a file is deleted on a computer, the information inside is never removed from the disk, and even the file name entry is left intact; the space and the file name entry used by the file are simply marked as “free to use” again, and until your operating system decides to overwrite that information with new stuff in the future, you must assume it’s all still there. In fact, software such as EnCase exists mainly for the purpose of finding such data so that they can use it against you.

There is no doubt that if you’re reading this, you want to destroy all possible traces of anything on your hard drive. It’s conceivable that a partially overwritten file name entry whose ending after the overwritten first four characters is “normal.avi” could be used against you if an allegedly infringed movie is the film “Paranormal.” The only way to remove data from a hard drive so that no one short of the CIA or FBI could ever hope to recover anything is to write something different to every single sector of the drive. That means backing up your important information to a different drive or CD/DVD, making sure you have the required discs or installers and license information to reinstall every single thing on your system, running a drive wiping tool that performs the complete erasure as described, and reinstalling everything from scratch. If you aren’t the most computer-literate person or you’re not confident in your abilities to get this done, seek the assistance of a professional computer service provider that can understand this paragraph’s requirements clearly. If you previously stored other sensitive information on the system, a drive wipe will ensure that those deleted files are definitely destroyed and not recoverable.

At a minimum, you need to have a product recovery CD or DVD that came with your computer system handy, along with a license key (usually on a sticker applied to the side of the computer system, consisting of five groups of five characters each, separated by hyphens.) It is highly likely that you’ll need a separate driver CD if your computer came with a “real” Windows XP, Vista, or 7 installation disc instead of a “product recovery” style disc that restores the system to its original state as shipped from the manufacturer. Some computers don’t come with discs and you must use a program included with the system to burn your own copies of the recovery media. Contact your computer manufacturer if you cannot find any of these things or you don’t have the discs required. Alternatively, call a computer service provider, especially if you’re starting to feel uncomfortable about the process already.

The easiest way to perform a full hard drive wipe yourself is to download and run Darik’s Boot and Nuke on your machine. DBAN WILL ERASE EVERY DRIVE CONNECTED TO YOUR COMPUTER, SO UNPLUG ALL EXTERNAL HARD DRIVES, FLASH DRIVES, AND CAMERA PICTURE CARDS BEFORE RUNNING IT. If you need help with using DBAN, Google is your friend; chances are that your questions will be answered with a simple search. You MUST WIPE YOUR HARD DRIVE ENTIRELY whether with DBAN or another tool made for that purpose, otherwise the latent information you’re concerned with WILL NOT BE ERASED, and you will remain at risk! Deletion and Windows-based “wipe” or “shred” or “secure delete” tools such as CCleaner ARE NOT SUFFICIENT TO ENSURE ALL DATA IS ELIMINATED!

YOUR COMPUTER IS NOT THE ONLY TARGET: Even if you wipe your computer’s hard drive entirely, information may exist on other media that could be subject to subpoena. Once your property is subpoenaed in a court case, you can be charged with a serious criminal offense if you tamper with any potential evidence, so performing these steps as soon as possible is vital to damage control. Consider any data that is stored on external hard drives, CD/DVD, USB flash drives, flash cards, cell phones, portable music players, etc. as potential targets. If in doubt, back up all of the information you care about to an external hard drive and put that drive somewhere safe (and preferably not somewhere you control such as a safe deposit box or your home or office cubicle!) Then proceed to destroy the data on these devices as needed. The previously mentioned CCleaner program has a “wipe free space” tool which may get the job done; I’ve personally used it but I haven’t analyzed a CCleaner wipe on a forensic level to ensure it actually wipes enough traces, so don’t assume it will take care of it. If in doubt, plug all of the devices you’re worried about into the machine before you use DBAN, and have them all wiped out at that point in time. Optical discs such as CDs and DVDs can be destroyed by tossing them one at a time into a microwave and running the microwave for approximately 8 seconds, which will cause the recording layer of the media to arc and disintegrate, effectively rendering it unreadable without special equipment and many man-hours of effort.

DON’T INSTALL SOFTWARE OR VISIT SITES USED FOR INFRINGEMENT: That means staying away from Ares, FrostWire, LimeWire, uTorrent, and anything similar. The mere existence of any such program on your computer (or the entries they make in the system registry that show they were once installed) can be used as supporting evidence to find you guilty of infringement! Also avoid visiting websites commonly associated with infringement such as The Pirate Bay, IsoHunt, etc. as these sites appearing in your history could also be presented as evidence against you, and while it’s very weak evidence alone, the combination of The Pirate Bay in the browser history plus an installation of uTorrent combined are far more powerful evidence that you were potentially infringing than if only one of those is discovered. Visiting forums such as the PhoenixLabs forums (which hosts PeerGuardian2, but also hosts discussion about issues related to copyrights and infringement) should not be problematic, because such forums are often visited by individuals who are interested in understanding copyright and copyright lawsuits in greater depth, and you have a right to research the claims being made against you.

DON’T INFRINGE FROM NOW ON, AND EDUCATE OTHERS IN YOUR HOME: Many copyright lawsuits involve parents, grandparents, or other adults who did not infringe upon copyrights, only to discover that a child or other household member was responsible. Minors who infringe will bring threats and possible lawsuits upon their parents or guardians, so it is ABSOLUTELY CRUCIAL that you talk to everyone who uses your Internet service at your home or business about the copyright issue, and ensure that they are educated on how to avoid infringing in the first place. If you don’t infringe on copyright, you have less risk over your head! Make sure that you learn as much as you possibly can on how infringement happens so that you can be certain it doesn’t happen at your IP address. If you use Wi-Fi, you also need to talk to anyone who leeches off of your wireless connection, because their actions can bring the torrent police upon you also, and the courts don’t seem to go with the “I have open Wi-Fi and other people can get on it freely!” excuse too well. Education is key. Protect yourself with knowledge. A good place to start might be the Electronic Frontier Foundation.

Additionally, it never hurts to know your rights, regardless of how far things go. The EFF has produced a very helpful guide on search, seizure, and subpoenas that everyone should read at least once.


LIST OF COMPANIES THAT THREATEN AND SUE

The companies that engage in these actions should be made known for their reprehensible behavior, as well as the companies that pay them for such “services.” What follows is a (likely incomplete) list of companies that are known to have some degree of involvement with these malicious actions. Don’t give them another cent, so as to discourage this type of behavior, until they officially state that they will no longer engage in hostile actions against their own customers and potential customers. Links are provided where possible as references, so that you can research each company’s individual behavior easily. Companies like Takedown Piracy are a much better role model for how anti-piracy efforts should be conducted; they don’t threaten to sue unless a settlement is paid, and they don’t seem to go after individual infringers on peer-to-peer networks. They submit DMCA takedown requests to lock down the sources of pirated works in search results and shut down blogs and other sites whose sole purpose is to offer up pirated material. I don’t necessarily agree with 100% of their tactics or the attitude that Nate displays in a couple of his posts there, but they seem to have taken a more positive path than the “pay up or else” crowd.

Copyright Enforcement Group, LLC


I am not a lawyer and this is merely advice from a layperson within the information technology industry.

I will add more information and links here over time, to assist people with copyright infringement notice issues in researching and understanding what they’re up against. Feel free to check back anytime.

Advertisements

If you haven’t seen it yet, take a look at my personal website. I also have a YouTube channel with short films, computer builds, and photo/video production stuff that you might find interesting.

UPDATE: Someone thought it’d be funny to submit this to Hacker News. It looks like I made some ZFS fans pretty unhappy. I’ll address some of the retorts posted on HN that didn’t consist of name-calling and personal attacks at the end of this article. And sorry, “OpenZFSonLinux,” I didn’t “delete the article after you rebuked what it said” as you so proudly posted; what I did was lock the post to private viewing while I added my responses, a process that doesn’t happen quickly when 33 of them exist. It’s good to know you’re stalking my posts though. It’s also interesting that you appear to have created a Hacker News user account solely for the purpose of said gloating. If this post has hurt your feelings that badly then you’re probably the kind of person it was written for.

It should also be noted that this is an indirect response to advice seen handed out on Reddit, Stack Overflow, and similar sites. For the grasping-at-straws-to-discredit-me HN nerds that can’t help but harp on the fact that “ZFS doesn’t use CRCs [therefore the author of this post is incompetent],” would you please feel free to tell that to all the people that say “CRC” when discussing ZFS? Language is made to communicate things and if I said “fletcher4” or “SHA256” they may not know what I’m talking about and think I’m the one who is clueless. Damned if you do, damned if you don’t.


tl;dr: Hard drives already do this, the risks of loss are astronomically low, ZFS is useless for many common data loss scenarios, start backing your data up you lazy bastards, and RAID-5 is not as bad as you think.


Bit rot just doesn’t work that way.

I am absolutely sick and tired of people in forums hailing ZFS (and sometimes btrfs which shares similar “advanced” features) as some sort of magical way to make all your data inconveniences go away. If you were to read the ravings of ZFS fanboys, you’d come away thinking that the only thing ZFS won’t do is install kitchen cabinets for you and that RAID-Z is the Holy Grail of ways to organize files on a pile of spinning rust platters.

In reality, the way that ZFS is spoken of by the common Unix-like OS user shows a gross lack of understanding of how things really work under the hood. It’s like the “knowledge” that you’re supposed to discharge a battery as completely as possible before charging it again which hasn’t gone away even though that was accurate for old Ni-Cd battery chemistry and will destroy your laptop or cell phone lithium-ion cells far faster than if you’d have just left it on the charger all the time. Bad knowledge that has spread widely tends to have a very hard time dying. This post shall serve as all of the nails AND the coffin for the ZFS and btrfs feature-worshiping nonsense we see today.

Side note: in case you don’t already know, “bit rot” is the phenomenon where data on a storage medium gets damaged because of that medium “breaking down” over time naturally. Remember those old floppies you used to store your photos on and how you’d get read errors on a lot of them ten years later? That’s sort of like how bit rot works, except bit rot is a lot scarier because it supposedly goes undetected, silently destroying your data and you don’t ever find out until it’s too late and even your backups are corrupted.

“ZFS has CRCs for data integrity”

A certain category of people are terrified of the techno-bogeyman named “bit rot.” These people think that a movie file not playing back or a picture getting mangled is caused by data on hard drives “rotting” over time without any warning. The magical remedy they use to combat this today is the holy CRC, or “cyclic redundancy check.” It’s a certain family of hash algorithms that produce a magic number that will always be the same if the data used to generate it is the same every time.

This is, by far, the number one pain in the ass statement out of the classic ZFS fanboy’s mouth and is the basis for most of the assertions that ZFS “protects your data” or “guards against bit rot” or other similar claims. While it is true that keeping a hash of a chunk of data will tell you if that data is damaged or not, the filesystem CRCs are an unnecessary and redundant waste of space and their usefulness is greatly over-exaggerated by hordes of ZFS fanatics.

Hard drives already do it better

Enter error-correcting codes (ECC.) You might recognize that term because it’s also the specification for a type of RAM module that has extra bits for error checking and correction. What the CRC Jesus clan don’t seem to realize is that all hard drives since the IDE interface became popular in the 1990s have ECC built into their design and every single bit of information stored on the drive is both protected by it and transparently rescued by it once in a while.

Hard drives (as well as solid-state drives) use an error-correcting code to protect against small numbers of bit flips by both detecting and correcting them. If too many bits flip or the flips happen in a very specific way, the ECC in hard drives will either detect an uncorrectable error and indicate this to the computer or the ECC will be thwarted and “rotten” data will successfully be passed back to the computer as if it was legitimate. The latter scenario is the only bit rot that can happen on the physical medium and pass unnoticed, but what did it take to get there? One bit flip will easily be detected and corrected, so we’re talking about a scenario where multiple bit flips happen in close proximity and in such a manner that it is still mathematically valid.

While it is a possible scenario, it is also very unlikely. A drive that has this many bit errors in close proximity is likely to be failing and the the S.M.A.R.T. status should indicate a higher reallocated sectors count or even worse when this sort of failure is going on. If you’re monitoring your drive’s S.M.A.R.T. status (as you should be) and it starts deteriorating, replace the drive!

Flipping off your CRCs

Note that in most of these bit-flip scenarios, the drive transparently fixes everything and the computer never hears a peep about it. ZFS CRCs won’t change anything if the drive can recover from the error. If the drive can’t recover and sends back the dreaded uncorrectable error (UNC) for the requested sector(s), the drive’s error detection has already done the job that the ZFS CRCs are supposed to do; namely, the damage was detected and reported.

What about the very unlikely scenario where several bits flip in a specific way that thwarts the hard drive’s ECC? This is the only scenario where the hard drive would lose data silently, therefore it’s also the only bit rot scenario that ZFS CRCs can help with. ZFS with CRC checking will detect the damage despite the drive failing to do so and the damage can be handled by the OS appropriately…but what has this gained us? Unless you’re using specific kinds of RAID with ZFS or have an external backup you can restore from, it won’t save your data, it’ll just tell you that the data has been damaged and you’re out of luck.

Hardware failure will kill your data

If your drive’s on-board controller hardware, your data cable, your power supply, your chipset with your hard drive interface inside, your RAM’s physical slot connection, or any other piece of the hardware chain that goes from the physical platters to the CPU have some sort of problem, your data will be damaged. It should be noted that SATA drive interfaces use IEEE 802.3 CRCs so the transmission from the drive CPU to the host system’s drive controller is protected from transmission errors. Using ECC RAM only helps with errors in the RAM itself, but data can become corrupted while being shuffled around in other circuits and the damaged values stored in ECC RAM will be “correct” as far as the ECC RAM is concerned.

The magic CRCs I keep making fun of will help with these failures a little more because the hard drive’s ECC no longer protects the data once the data is outside of a CRC/ECC capable intermediate storage location. This is the only remotely likely scenario that I can think of which would make ZFS CRCs beneficial.

…but again: how likely is this sort of hardware failure to happen without the state of something else in the machine being trashed and crashing something? What are the chances of your chipset scrambling the data only while the other millions of transistors and capacitors on the die remain in a functional and valid working state? As far as I’m concerned, not very likely.

Data loss due to user error, software bugs, kernel crashes, or power supply issues usually won’t be caught by ZFS CRCs at all. Snapshots may help, but they depend on the damage being caught before the snapshot of the good data is removed. If you save something and come back six months later and find it’s damaged, your snapshots might just contain a few months with the damaged file and the good copy was lost a long time ago. ZFS might help you a little, but it’s still no magic bullet.

Nothing replaces backups

By now, you’re probably realizing something about the data CRC gimmick: it doesn’t hold much value for data integrity and it’s only useful for detecting damage, not correcting it and recovering good data. You should always back up any data that is important to you. You should always keep it on a separate physical medium that is ideally not attached to the computer on a regular basis.

Back up your data. I don’t care about your choice of filesystem or what magic software you write that will check your data for integrity. Do backups regularly and make sure the backups actually work.

In all of my systems, I use the far less exciting XFS on Linux with metadata CRCs (once they were added to XFS) on top of a software RAID-5 array. I also keep external backups of all systems updated on a weekly basis. I run S.M.A.R.T. long tests on all drives monthly (including the backups) and about once a year I will test my backups against my data with a tool like rsync that has a checksum-based matching option to see if something has “rotted” over time.

All of my data loss tends to come from poorly typed ‘rm’ commands. I have yet to encounter a failure mode that I could not bounce back from in the past 10 years. ZFS and btrfs are complex filesystems with a few good things going for them, but XFS is simple, stable, and all of the concerning data loss bugs were ironed out a long time ago. It scales well and it performs better all-around than any other filesystem I’ve ever tested. I see no reason to move to ZFS and I strongly question the benefit of catching a highly unlikely set of bit damage scenarios in exchange for the performance hit and increased management complexity that these advanced features will cost me…and if I’m going to turn those features off, why switch in the first place?


Bonus: RAID-5 is not dead, stop saying it is

A related category of blind zealot is the RAID zealot, often following in the footsteps of the ZFS zealot or even occupying the same meat-suit. They loudly scream about the benefits of RAID-6, RAID-10, and fancier RAID configurations. They scorn RAID-5 for having terrible rebuild times, hype up the fact that “if a second drive dies while rebuilding, you lose everything!” They point at 10TB hard drives and do back-of-the-napkin equations and tell you about how dangerous and stupid it is to use RAID-5 and how their system that gives you less space on more drives is so much better.

Stop it, fanboys. You’re dead wrong and you’re showing your ignorance of good basic system administration practices.

I will concede that your fundamental points are mostly correct. Yes, RAID-5 can potentially have a longer rebuild time than multi-stripe redundant formats like RAID-6. Yes, losing a second drive after one fails or during a rebuild will lose everything on the array. Yes, a 32TB RAID-5 with five 8TB drives will take a long time to rebuild (about 50 hours at 180 MB/sec.) No, this isn’t acceptable in an enterprise server environment. Yes, the infamous RAID-5 write hole (where a stripe and its parity aren’t both updated before a crash or power failure and the data is damaged as as result) is a problem, though a very rare one to encounter in the real world. How do I, the smug techno-weenie advocating for dead old stupid RAID-5, counter these obviously correct points?

  • Longer rebuild time? This is only true if you’re using the drives for something other than rebuilding while it’s rebuilding. What you really mean is that rebuilding slows down less when you interrupt it with other work if you’re using RAID levels with more redundancy. No RAID exists that doesn’t slow down when rebuilding. If you don’t use it much during the rebuild, it’ll go a lot faster. No surprise there!
  • Losing a second drive? This is possible but statistically very unlikely. However, let’s assume you ordered a bunch of bad Seagates from the same lot number and you really do have a second failure during rebuild. So what? You should be backing up the data to an external backup, in which case this failure does not matter. RAID-6 doesn’t mean you can skip the backups. Are you really not backing up your array? What’s wrong with you?
  • RAID-5 in the enterprise? Yeah, that’s pretty much dead because of the rebuild process slowdown being worse. An enterprise might have 28 drives in a RAID-10 because it’s faster in all respects. Most of us aren’t an enterprise and can’t afford 28 drives in the first place. It’s important to distinguish between the guy building a storage server for a rack in a huge datacenter and the guy building a home server for video editing work (which happens to be my most demanding use case.
  • The RAID-5 “write hole?” Use an uninterruptible power supply (UPS). You should be doing this on any machine with important data on it anyway! Assuming you don’t use a UPS, Linux as of kernel version 4.4 has added journaling features for RAID arrays in an effort to close the RAID-5 write hole problem.

A home or small business user is better off with RAID-5 if they’re also doing backups like everyone should anyway. With a 7200 RPM 3TB drive (the best $/GB ratio in 7200 RPM drives as of this writing) costing around $95 each shipped, I can only afford so many drives. I know that I need at least three for a RAID-5 and I need double as many because I need to back that RAID-5 up, ideally to another machine with another identically sized RAID-5 inside. That’s a minimum of six drives for $570 to get two 6TB RAID-5 arrays, one main and one backup. I can buy a nice laptop or even build a great budget gaming desktop for that price, but for these storage servers I haven’t even bought the other components yet. To get 6TB in a RAID-6 or RAID-10 configuration, I’ll need four drives instead of three for each array, adding $190 to the initial storage drive costs. I’d rather spend that money on the other parts and in the rare instance that I must rebuild the array I can use the backup server to read from to reduce my rebuild time impact. I’m not worried about a few extra hours of rebuild.

Not everyone has thousands of dollars to allocate to their storage arrays or the same priorities. All system architecture decisions are trade-offs and some people are better served with RAID-5. I am happy to say, however, that if you’re so adamant that I shouldn’t use RAID-5 and should upgrade to your RAID levels, I will be happy to take your advice on one condition.

Buy me the drives with your own money and no strings attached. I will humbly and graciously accept your gift and thank you for your contribution to my technical evolution.

If you can add to the conversation, please feel free to comment. I want to hear your thoughts. Comments are moderated but I try to approve them quickly.


Update to address Hacker News respondents

First off, it seems that several Hacker News comments either didn’t read what I wrote, missed a few things, or read more into it than what I really said. I want to respond to some of the common themes that emerged in a general fashion rather than individually.

I am well aware that ZFS doesn’t exactly use “CRCs” but that’s how a lot of people refer to the error-checking data in ZFS colloquially so that’s the language I adopted; you pointing out that it’s XYZ algorithm or “technically not a CRC” doesn’t address anything that I said…it’s just mental masturbation to make yourself feel superior and it contributes nothing to the discussion.

I was repeatedly scolded for saying that the ZFS checksum feature is useless despite never saying that. I acknowledge that it does serve a purpose and use cases exist. My position is that I believe ZFS checksums constitute a lot of additional computational effort to protect against a few very unlikely hardware errors once the built-in error checking and correction in most modern hardware is removed from the overall picture. I used the word “most” in my “ZFS is useless for many common data loss scenarios” statement for a reason. This glossing over of important details is the reason I refer to such people as ZFS “zealots” or “fanboys.” Rather than taking the time to understand my position fully before responding, they quickly scanned the post for ways to demonstrate my clear ignorance of the magic of ZFS to the world and jumped all over the first thing that stood out.

 

kabdib related an anecdote where the RAM on a hard drive’s circuit board was flipping data bits in the cache portion and that the system involved used an integrity check similar to ZFS which is how the damage was detected. The last line sums up the main point: “Just asserting “CRCs are useless” is putting a lot of trust on stuff that has real-world failure modes. Remember that I didn’t assert that CRCs are useless; I specifically outlined where the ZFS checksum feature cannot be any more helpful than existing hardware integrity checks which is not the same thing. I question how common it is for hard drive RAM to flip only the bits in a data buffer/cache area without corrupting other parts of RAM that would cause the drive’s built-in software to fail. I’m willing to bet that there aren’t any statistics out there on such a thing. It’s good that a ZFS-like construct caught your hardware issue, but your obscure hard drive failure anecdote does not necessarily extrapolate out to cover billions of hard drives. Still, if you’re making an embedded device like a video game system and you can afford to add that layer of paranoia to it, I don’t think that’s a bad thing. Remember that the purpose of my post is to address those who blindly advocate ZFS as if it’s the blood of Computer Jesus and magically solves the problems of data integrity and bit rot.

rgbrenner offered indirect anecdotal evidence, repetitions of the lie that I asserted “CRCs are useless,” and then made a ridiculous attempt at insulting me: “If this guy wrote a filesystem (something that he pretends to have enough experience to critique), it would be an unreliable unusable piece of crap. Well then, “rgbrenner,” all I can say is that if you are so damned smart and have proof of this “unreliable and unusable” state that it’s in, file a bug against the filesystem I wrote and use on a daily basis for actual work so it can be fixed, and feel free to keep the condescending know-it-all attitude to yourself when you do so.

AstralStorm made a good point that I’ve also been trying to make: if your data is damaged in RAM that’s not used by ZFS, perhaps while the data is being edited in a program, it can be damaged while in RAM and ZFS will have no idea that it happened.

wyoung2 contributed a lot of information that was well-written and helpful. I don’t think I need to add anything to it, but it deserves some recognition since it’s a shining chunk of gold in this particular comment septic tank.

X86BSD said that “Consumer hardware is notriously busted. Even most of the enterprise hardware isn’t flawless. Firmware bugs etc. .” I disagree. In my experience the vast majority of hardware works as expected. Even most of the computers with every CPU regulator capacitor leaking electrolyte pass extended memory testing and CPU burn-in tests. Hard drives fail a lot more than other hardware does, sure, but even then the ECC does what it’s supposed to do and detects the error and reports it instead of handing over the broken data that failed the error check. I’d like some hard stats rather than anecdotes but I’m not even sure if they exist due to the huge diversity of failure scenarios that can come about.

asveikau recalls the hard drive random bit flipping problem hitting him as well. I don’t think that this anecdote has value because it’s a hard drive hardware failure. Sure, ZFS can catch it, but let’s remember that any filesystem would catch it because the filesystem metadata blocks will be read back with corruption too. XFS has optional metadata CRCs and those would catch this kind of disk failure so I don’t think ZFS can be considered much better for this failure scenario.

wyoung2 made another lengthy comment that requires me to add some details: I generally work only in the context of Linux md RAID (the raid5 driver specifically) so yes, there is a way to scrub the entire array: ‘echo check > /sys/block/md0/md/sync_action’. Also, if a Linux md RAID encounters a read error on a physical disk, the data is pulled from the remaining disk(s) and written back to the bad block, forcing the drive to either rewrite the data successfully or reallocate the sector which has the same effect; it no longer dumps a whole drive from the RAID on the basis of a single read error unless the attempts to do a “repair write” fail also. I can’t really comment on the anecdotal hardware problems discussed; I personally would not tolerate hardware that is faulty as described and would go well out of my way to fix the problem or replace the whole machine if no end was in sight. (I suppose this is a good time to mention that power supply issues and problems with power regulation can corrupt data…)

Yet another wyoung2 comment points out one big advantage ZFS has: if you use RAID that ZFS is aware of, ZFS checksums allow ZFS to know what block is actually bad when you check the array integrity. I actually mentioned this in my original post when I referenced RAID that ZFS pairs with. If you use a proper ZFS RAID setup then ZFS checksums become useful for data integrity; my focus was on the fact that without this ZFS-specific RAID setup the “ZFS protects your data” bullet-point is false. ZFS by itself can only tell you about corruption and it’s a dangerous thing to make people think the protection offered by a ZFS RAID setup is offered by ZFS by itself.

At this point I can only assume that rgbrenner just enjoys being a dick. And that, in contrast, AstralStorm understood what I was trying to say to at least some extent.

DiabloD3 quoted me on “RAID is not a replacement for backups” and then mentions ZFS external backup commands. Hey, uh, you realize that the RAID part was basically a separate post, right? In fact, there is not a single mention of ZFS in the RAID section of the post other than as a topic transition mechanism in the first paragraph. I included the RAID part because the ZFS religion and the RAID-over-5-only religion have the same “smell.”

I’ll have to finish this part later. It takes a lot of time to respond to criticism. Stay tuned for more. I have to stop so I can unlock the post and keep OpenZFSonLinux from eating off his own hands with anticipation. As a cliff-hanger, check this out…I enjoyed the stupidity of X86BSD’s second comment about me endangering my customers’ data [implicitly because I don’t use ZFS] so much that I changed my blog to integrate it and embrace how horrible of a person I am for not making my customers use ZFS with checksums on their single-disk Windows machines. If my destiny is to be “highly unethical” then I might as well embrace it.

%d bloggers like this: