Index ⇒ General Topics ⇒ Storage/backup solutions for large data

[Googler123]

Hello everyone,

I'm wondering here, I've gathered a whole lot of data lately. Games and rare cartoons mostly but also some rare tv shows. I'm imagining being game collectors you've had some ideas about storage. I have 2*tb of external hdds but those are almost full.
A hdd of an older laptop, which I used to store small games crashed on me a few weeks ago, losing 300+ titles in the process (most of them are still findable though but you really have to dig).
I've considered getting a blu ray drive and using blu rays. Cumbersome and obsolete tech, but supposedly should last more than dvds.
HDDs are much easy to use, but like I said can crash on you leading to data loss. And all of it it's gone. Just gone.
So how do you do it? what do you use to store your loots?

[hgdagon]

Well, Baidu offers 2TB for free, but in my country, for example, the speed is brutally low, if it's OK in your country, you might want to give it a try. You may need to use Google translator to be able to sign up. If you're up to paid solution, I would recommend MEGA.

[Shattered]

There's a new service in my country. They offer 1000 GB for free, but you need to enlist so no idea when it will be available.

Code: Select all

https://www.transip.nl/stack/

I'll let you know how it is once I have access to it.

[hfric]

Crashed how ? every HDD data is recoverable
But you have to take apart the HDD
sometimes its the MBR that's corrupt http://www.partition-tool.com/easeus-pa ... ld-mbr.htm
you could also do http://hddguru.com/software/2005.10.02-MHDD/ for diagnose and repair

[Googler123]

The hdd crashed mechanically. I tried switching the circuit board with one of the exact serial number. Didn't work. Professional data restore costs a lot. The data was replaceable. So won't try a recovery. I already had some of the things backed in cloud, the others are pretty easy to get, but I won't bother.
I had them indexed with a program .
Online backup is good. There's little chance of failure, but if it's not paid many services delete your files. Interesting stuff about baidu, but they must be overadvertising their service. I'd say try to put 500GB in there and see if they start deleting.

It's best to use a shotgun approach: use multiple services. But then managing your backups becomes tedious.

[Meddle]

I have more than 6TB backed up on DVDs. Note that this takes up a good bit of space. It's nearly 1,300 discs.

[Googler123]

Whoa meddle you sound like my kind of guy. I trust it you have indexes of what is where, otherwise it would be a nightmare. I index my things with disc explorer professional; probably a txt or excel file of some kind would also work. I also have my old collection of ~300 dvd-s, but they couldn't match the convenience of external HDD-s. Also some of them 2-3% were unreadeable when they were transfered to the HDD.
Have you considered blu-ray? They seem dirt cheap, the only initial investment would be the drive, maybe 80$, and the discs (where I live they're about 1$ a pop, but I've seen cheapear. Alas might be of poor quality and cause data loss)
let's say 20gb per disk, 5 discs per 100gb, 50 discs per TB, ~ 300 discs for your collection. But transferring all that... uh

[Meddle]

I'm using WhereIsIt for my indexing.

I also don't have any money to go toward a blu-ray burner, discs for it or hdd's. I prefer to price per gigabyte; not whether the disc is $1 to $2 or more/less. When I go that route I'm going to use Verbatim DataLifePlus 50GB's and hdd's. But not the external usb ones. Those use cheap mechanical laptop drives in them. But all my discs are still readable even though some of them are pushing 10 years of age now.

If you wanted to haul around a hdd it would be best to have an external enclosure with an ssd in it.

[Googler123]

Whereisit seems a neat little program, thanks for the tip
I use hdds with a separate power cable, 3.5 ", not exactly portable but they get the job done. I don't move them around except on rare ocasions so yeah. I've heard portable external drives are notoriously bad, with high failure rates.
Regarding dvd-s: they have a different dye as opposed to the bluray. It's organic and doesn't last that long supposedly. But it differs by manufacturer. I really think the bluray is more efficient in $/GB. Where I am they're ~ 2-4 more expensive than dvds, but they hold 4-5 times the data.
The cost of the drive would be prohibitive though. $/gb it starts overthrowing hdds at 5TB mark.
The problem with discs is you can't update the data on them. For instance if you have a new release you can't redo them.

[Swimmin]

To me, it is important to know what I need/expect and classify accordingly: Am I looking for
- access speed
- archive quality
- temporary/short term storage
- size of data
- etc. ??

But what is most important is to decide which data can be lost (but then.. why (spend money/effort to) save it?) and which data we really actually want to save..

Most solutions have their problems:

• Generic writable optical media (organic dye degrades over time, bit-rot, no guarantee you retrieve what you stored)
• Conventional hd (can break, bit-rot (although, interestingly, numerous data-recovery experts state they've never seen actual bit-rot and the advent of ZFS later proved 'bitrot' on HD's usually was faulty data from (s)ata/ide controllers and bit-rot in memory))
• Tape (somewhat less 'conventional' to an average user, bitrot, more constrained storage requirements)
• Flash (more expensive compared to generic writable optical media, bit-rot)
• Cloud (privacy, even a monetary consolation 'prize' when they loose your child-hood video's just doesn't compensate the loss)

All above need some form of error-checking (to verify if you got what you stored).
Often people duplicate the media completely (which doubles cost, and still leaves headaches once one tries to restore). Sometimes people add parity files as well (bloating size).
All of this is just hassle (especially with very large files) and extra cost and ideally we'd like to avoid that.

In my current opinion, that leaves just 2 solutions (for long(er) term data storage where we try to actually retrieve the data exactly as we once stored it:

• Home-Fileserver running ZFS (file system) cluster on a 64bit machine with ECC support and at least 8GiB ECC Memory (2 disk spares).
  Con: growing your pool is a bitch/impossible (depending on implementation), price (as you start with as much overcapacity as you can afford, also talking about spares and future identical replacements). Can't (cheaply) store duplicate in relative/friend's home.
  Pro: fast access-time (over network), safe against bitrot; as long as your set works you got what you stored, no excuses.
  
• M-Disc (available in DVD 4.7, BD-R 25, BD-R 50 and BD XL 100)
  Con: slower access (and you need some indexing software). Requires a working drive (in the future)
  Price per disk drops when you realize you don't need to copy your archives every x-years and don't require duplicates (compared to conventional optical media)
  Pro: safe against bitrot, you get what you verified after you burned it, no excuses for >lifetime. Cheaper to start: get M-disc compatible drive (archival quality optional), enclose in decent usb-3 enclosure, store with disks and buy the discs. Also, compared to the zfs file-server, this does what most of the people envisioned they were doing when they en-masse transferring their childhood video's to DVD (and tossing the degrading original tapes). Finally, no virus can modify the data after storage... Price-efficient to store a duplicate at a friend/relative's home
  Your current 2TB could fit on just 20 BD XL 100 discs...

No solutions (sadly) are 'for-ever' not even 'our life-time' (due to evolution in devices and their interfaces). At some point one will transfer the M-disc's to a medium more appropriate to the time, make no illusions about that. However, the beauty is that once the time comes that we know almost nobody has an USB3 port anymore (and almost no converters and/or BD Drives) and we decide to migrate our data to another medium, we still get the data we once stored! Besides, it is not unreasonable that a specialized digital archival/forensic lab in the future will still have a specialized optical drive to recover data with..

Personally I think M-Disk can give a time-span of 20 to 25 years (specifically: before it is no longer reasonably possible to obtain a PC with USB3 Compatible (adapter) interface and (external) BD Drive secondhand without paying ludicrous prices)

Just my 2 cents

[myloch]

I'd like to upload a game that might fit in the site but it's quite big, so it would fill the majority of 4shared account space, anyone with a lot of free space or other solutions? Mega?

[Meddle]

Mega gives 50GB for registered free users.

[Shattered]

I'm seriously considering Usenet as a place to "backup" stuff. Some servers offer more than 7 years retention.

[Meddle]

There are a few groups on Usenet that feature posts comprised of files both passworded and obfuscated. Based on those observations I'm fairly sure that some people have been doing exactly that for several years.

[Tadrio]

There are a few cloud backup providers that offer "unlimited" space for very low prices. I personally use Crashplan and Backblaze. They both set you back around $5-$6 a month. I signed up for both because I wanted to compare and pick one, but kept the two because I'm really happy with both, and I figure it's an additional level of safety.

Both of them offer private key encryption, so your data is encrypted and decrypted locally, with a key that only you possess. Although it should be noted that both their clients are proprietary and closed-source, so you'll have to take their word for it. What's nice about both is that they just run, index, deduplicate, encrypt and upload in the background, all of your complete harddrives if you want, so you really don't have to care after setting it up, unless you end up needing a recovery. It feels nice that now if my computer exploded I would no longer be worried about the data, only the hardware costs.

I don't know how their unlimited storage works out in practice, but I have stored over 7 TB with both of them now, neither one complained or throttled my upload speed. If you're outside of North America though, Backblaze is the better choice, because uploading from overseas is much faster than with Crashplan. They're talking about adding European servers, but nothing yet, and I assume I'd have to re-upload the whole 7TB if I wanted to switch.

There are some minor differences to the services. Crashplan has a Java-based client that you'll have to tell which folders or drives to back up, while Backblaze's client is native and backs up everything by default. Both offer versioning and a buffer in which they keep deleted files, although only Crashplan allows you to configure it so that it keeps versions and deleted files indefinitely. Backblaze removes files you deleted locally after 30 days. And both offer mobile apps to access your data. They're a bit cumbersome to use if you have a 448-bit private encryption key, but it's nice to be able to access a file you forgot at home. Backblaze has a pretty low download limit for the mobile client though, I think you can't load files over 30 MB.