Distributed Backups?
from the another-solutions dept
Having lost way too many hard-drives in the last couple years, I’ve become pretty focused on backup systems, and (judging from a few conversations here at DEMOmobile) it’s a subject that a lot of people are interested in. Bob Cringely’s latest column is another one of his “big idea for a company” suggestions that he simply tosses out and hopes someone builds: a peer-to-peer distributed backup system. The idea is pretty simple. You download some backup software which encrypts the data you want to backup, breaks it up into little chunks and moves it out over a peer-to-peer network. The amount of data you can backup is limited by the amount you agree to store yourself. In Cringely’s plan, this whole thing should cost $3.95/month. Of course, you’d need to distribute many copies of the various pieces to make sure that there aren’t chunks missing — which will also mean that many people will need to be in the system before it’s really all that reliable. Obviously, some people will freak out about chunks of their data (even if they are encrypted) sitting on random hard drives out there). In the meantime, it appears Cringely doesn’t use Google, because a quick search on the topic suggests at least some people are already working on this type of technology.
Comments on “Distributed Backups?”
Delta
Crucial to this technology will be the truly incremental file transfer: just sending those bytes which have changed. As files get bigger and bigger, it’s important to reduce the overall traffic.
No Subject Given
Nah, this would be a huge security risk, and hey at only 128Kbps upload, would not be nice even on DSL.
I’ve tried Ghost 7/2003 and Acronis TrueImage6, but DriveImage7 (now owned by Symantec, and seems to be worked into Ghost9) seems to be the nicest so far, but still a little buggy (it’s a known bug that if you tick the “checksum image when done” box, it will render the image useless).
I’d really like something that would work across Windows, and all the new journaled Linux filesystems, but haven’t found anything that will do more than FAT32/NTFS/ext3.
The latest trend is allowing you to do the backup in Windows – which means you can write the image to any device, as you have the drivers (not some stupid DOS boot floppy).
Someone needs to write something that will just copy whatever is on a HDD (regardless of format) to any device you want (DVD-RW, HDD, LAN) just with a simple boot CD. It must however retain things like the boot sector, MBR, and partition information (you ever tried to ghost a dual booting drive?!)
I guess the best way is to just get one of those hardware drive duplicators, but they cost thousands and you need two identical drives.
quote
“Backups are for wimps. Real men upload their data to an FTP site and have everyone else mirror it.”
– Linus Torvalds
Onion Networks (Justin Chapweske)
Justin Chapweske (who authored SwarmCast, a key component of OpenCola‘s late, lamented plan for world domination) has done a lot of work that could be (or may have already been) applied to this. He likes distributing things.
I know it’s probably something that most people know already, but just in case: it’s also worth digging into “forward error correction” a bit as a mechanism for offsetting the impact of the “ensure that you aren’t missing a key piece of your data” concerns.
You can buy this product already!
This product has been on the market for a while. Go to http://www.hivecache.com/. No, I am not connected with them.
Done deal
A student of mine developed a system to do this a couple of years ago. All encrypted and redundant so that missing machines were not a problem. He won “best student project in the UK” for it in fact. There are commercial versions available too, though not with quite the technical sophistication that our system has (AFAIK).
Corporate Intelligence?
Jesus… hive cache has been arond so long I don’t really know how Cringely or Tech Dirt can sleep at night after trying to pass themselves off as credable tech pundits, let alone “Corporate Intelligence”.
Still, the reality of the matter is that the lead time that hive cache had will soon be comming to a close. First group to successfully build a nice, well integrated windows protduct that they can sell out to Veritas or any one of the 10+ other backup product makers out there will win and become the new industry standard in backup…
Of course, when encrypted P2P hits the streets with anonymous distribution of keys via DNS, the RIAA is going to get a nifty little introducton to how the internet routes around damage.
Re: Corporate Intelligence?
Hive Cache has been around so long… that all of their website pages read “coming soon” and give no other details. Horror or horrors that we hadn’t heard of it before.
Re: Re: Corporate Intelligence?
Just because you weren’t invited to the party doesn’t mean that it didn’t happen.
private invitations for pre-release evaluation happened over a year ago.
dumbass…
Re: Re: Re: Corporate Intelligence?
Anonymous Troll/Coward/Idiot,
Wow! A pre-release evaluation over a year ago, and they have nothing new to offer on their web site. Really promising stuff!
Re: Re: Corporate Intelligence?
Mike, HiveCache has been around for *years*, first as p2p system “Mojo Nation” and later on as HiveCache, selling a distributed backup solution. The web site looks like it’s currently being renovated in preparation for some product announcements, but if you follow the p2p field at all you would have been familiar with these folks from a long way back.
Re: Re: Re: Corporate Intelligence?
We too are working on a P2P backup product StoreGrid, which is currently in Beta. Plan to release it around end of Jan 2005. StoreGrid 1.0 version is meant for use inside a corporate intranet. StoreGrid 1.0 Beta is available for download. Supports Encryption, Compression, Incremental Backup, Registry Backup, System Files Backup, Continuous Backup, Quick Restore etc. Works on Windows and Linux. Mac OS X will be supported in a few months.
Check out
http://www.vembu.com/storegrid/desktop-backup.html
P2P Backup
And these people too: LeanOnMe by 312, Inc.
P2P Backup
And these people too: LeanOnMe by 312, Inc.
No Subject Given
I’ve actually been working on something like this for awhile and you are welcome to download it at http://www.pensamos.com/mmb/ Version 1.0 is free. The peer network exists within your office, so you are just sharing with other computers in your office and your data therefore doesn’t leave your network unless you want it to.
There is also an optional off-site component to this, so that you can additionally backup to a remote server to protect against office wide failures like fire, theft, or hurricanes. Your backups are still encrypted, so the data is still secure. Present plans are to run the off-site servers for this as a paid service (with the p2p backup remaining free at least for version 1.0). However the server software is written in PHP and can easily be run on any commodity $3/month web-hosting service, and I might consider just releasing the server code for people to use on their own if there is an interest in that.
I would greatly appreciate any and all feedback on this program. I really want this program to evolve based on what users out there want, and I realize that the current version doesn’t have every ideal feature, so I would love to hear what people would like to have added first. You can send me feedback at twm@pensamos.com
Thanks,
– Tim Macinta
Not exeactly the 3 bucks Cringley wants...
First and foremost, I am associated with 312 Inc. makers of LeanOnMe P2P Backup and remote access software. I am their marketing director. Now that I’ve said that… LeanOnMe is in it’s initial release version 1.0. Shoot me an email in order to receive a free key. vikki_dot_fraser_at_312inc_dot_com. and visit http://www.312inc.com/3_sol_prod_LOM.html for the smiling happy people page or http://www.312inc.com/tech.htm for a more technical view.
I made a system like this for a University project
Info and reports at: http://pages.cpsc.ucalgary.ca/~reeder/cpsc502/
It is extremely fault tolerant too since it uses correction codes.