Index ⇒ General Topics ⇒ Operation LW2Usenet - Backing up our uploads on newsgroups

Thoughts on life, the universe and everything else not covered in other categories.

Moderator: LW Moderator

Operation LW2Usenet - Backing up our uploads on newsgroups

Postby Molitor » Wed May 31, 2017 8:43 pm

TL;DR
LW uploads are being uploaded to newsgroups for backup purposes.
Click HERE and HERE for an introduction to Usenet.
Click HERE for two free providers.
Click HERE to find the NZBs.


Introduction & background
In a 2016 post, Shattered outlined the possibility of using newsgroups to back up our uploads. His test upload from September 2010 can be found here for reference.
The outstanding benefit of newsgroups is their retention, meaning how long the files remain available on the servers. Unlike cyberlockers, files posted on newsgroups do not "expire" depending on their popularity; and nowadays, many providers reach 3000+ days of retention, meaning you can access files posted 8 years ago --provided they haven't been DMCA'd meanwhile-- and that files uploaded now should remain available for the next 8 years --provided they aren't DMCA'd in the meantime--. Binary retention is still growing for most of the providers, on a daily basis.
What I'm looking forward to is to provide a mirror for all LW uploads - from the oldass early 2000s Rips to the latest ISOs.


The main downsides
Because it can't be perfect.
  • First major disappointment: Subscribing for a good Usenet access is not free. However it is usually very cheap. If you are a humble downloader, looking for a few MBs there and there, a free trial at a major provider, or the free plan at Free-Usenet (1 Mbit/s = ~140 KB/s) will leave you satisfied. If you need to leech some terabytes off newsgroups, a paying subscription is probably what you'll need.
  • Second drawback: all files meant to be backed up for LW2Usenet are repacked and passworded before they are posted to newsgroups. This implies that double-packed scene releases, for example, become triple-packed, hence you will sometimes get files insides archives inside archives inside archives, à la matrioshka. Lately I've been trying to avoid this by removing the first layer of compression before processing the files, however the practice is still subject to debate, I would like to hear your opinion on it.
    This also means that in order to get a specific file inside these encrypted archives, you need to download the whole set of archives. E.g, you want rld-btaa.r60 ? It's only 95MB, but you'll need to download the whole 8GB set and decompress it to do so. Which brings us to the third part:

What's the point then?
Providing a long-term, last-chance backup for uploaders, and a viable option for everyday downloaders.
In other words, if you are an uploader, this mirror will likely only be useful for you in case you loose files and wish to reupload them to a cyberlocker.
However, if you are a downloader, chances are that you chose to download off Usenet because the original upload was broken in some way - remember you will need to download the whole game.


Modus operandi
As soon as the files are downloaded off their original location, they are packed in 200MB RAR archives with a password. Then, 10% of PAR2 recovery volumes are created and the set of files --RAR + PAR2-- are uploaded to alt.binaries.archive.encrypted.
If the original download is broken or misses files, I upload a replacement instead when possible, e.g when it's a scene release or something easy to find - otherwise all I can do is skip it, report it and process it later if a reupload happens.
RARing, PARing and uploading is automated through a script, on a server - so the only cost here is time and CPU usage. The script can process 10, 100 or 300GB of data with scarcely any human interaction, and that's pretty convenient. However, PAR2 generation is long and takes the larger part of the process (I'm looking forward to set up a more efficient system, using Parpar instead of Par2cmd), and the recent bandwidth limits are slowing down the operations as well.
Anyway, there are two movements to this leeching process. The first part consists in mirroring the uploads #1 to #9999. In the meantime, I'm also processing uploads starting from #10000 in the hopes of catching up the latest uploads (#13480 and so on as I'm writing this).
User avatar
Molitor
3DSL Moderator
 
Posts: 1017
Joined: Mon Feb 27, 2012 6:53 pm
Location: France
Thanks: 156
Thanked: 443 times in 324 posts

The following user would like to thank Molitor for this post
Scaryfun, Shattered, UberLamer

Operation LW2Usenet - Backing up our uploads on newsgroups

Postby Molitor » Wed May 31, 2017 9:11 pm

Where to find them
For the uploads that have been already processed, you may find them on your favourite indexer:
Code: Select all
https://nzbindex.com/search/?poster=LW2Usenet+%3CLW2U%40Legendsworld.net%3E&more=1
https://binsearch.info/?server=2&max=250&g=alt.binaries.archive.encrypted&a=LW2Usenet+%3CLW2U%40Legendsworld.net%3E
http://www.nzbking.com/poster/LW2Usenet%20%3CLW2U%40Legendsworld.net%3E/

Alternate download location (with search feature!) (Randomly on and offline, depending on my mood)
Code: Select all
http://163.172.58.206:8080


The file sets are named after their LW upload number, and presented as such: LW_[mirror n°].
Ex: You want to download "Batman: Vengeance - AlcoholClone ISO". Notice in the URL how the mirror number is specified: "download.php?mirror=10100". So this upload is mirror #10100. Thus it can be found as "LW_10100" on your indexer.

All archives are protected with the password
LW
Keep in mind that even after unpacking these files with the LW password, the files inside may require a second password, provided in the Uploader's notes.
User avatar
Molitor
3DSL Moderator
 
Posts: 1017
Joined: Mon Feb 27, 2012 6:53 pm
Location: France
Thanks: 156
Thanked: 443 times in 324 posts

The following user would like to thank Molitor for this post
Shattered

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Scaryfun » Thu Jun 01, 2017 2:57 am

That's a massive undertaking. :thanks: for all your efforts. I'm doing an update tonight, but I'll check tomorrow. It's sad when we lose games that have been upped before. So, with this, it will prevent further losses.
Image
User avatar
Scaryfun
3DSL Admin
 
Posts: 9354
Joined: Wed Jun 02, 2004 9:27 pm
Location: Toronto, Canada.
Thanks: 2142
Thanked: 921 times in 805 posts

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Shattered » Thu Jun 01, 2017 6:25 pm

Great job!
User avatar
Shattered
3DSL Moderator
 
Posts: 2408
Joined: Tue Apr 08, 2008 10:03 pm
Thanks: 627
Thanked: 531 times in 412 posts

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Shattered » Sun Jul 02, 2017 11:06 am

XS Usenet has raised the retention to 1100 days and speed to 2 Mbit for free accounts, but with a 25 GB data limit (not sure if this is per month). Combined with a Free-Usenet account you can download posts up to 3 years old with a reasonable 350 kB/s speed without paying a cent.

XS Usenet

- Download speed: 2 Mbit/s
- Connections: 5
- 25 GB data limit
- Unlimited days
- No SSL
- 1100 days binary retention


Code: Select all
https://www.xsusenet.com/


Free-Usenet

- Download speed: 1 Mbit/s
- Connections: 1
- Unlimited traffic
- Unlimited days
- Free SSL included
- 2700+ days binary retention

Note:

A free account needs to be renewed every 24 hours, and the password changes with every renewal.


Code: Select all
https://free-usenet.com/
User avatar
Shattered
3DSL Moderator
 
Posts: 2408
Joined: Tue Apr 08, 2008 10:03 pm
Thanks: 627
Thanked: 531 times in 412 posts

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Molitor » Fri Jul 14, 2017 5:59 pm

:idea: Update on the progress after approx. 1 month 1/2 of running the service:
-Uploads #1-1500 are now covered, as well as #10000-10600.
-Many have been skipped, though: most of them are rips uploaded by the long-away-now MasteromaN. All of these rips have an ISO equivalent anyway, so they are not exactly high-priority stuff. There's a bit of backlog due to incomplete downloads awaiting to be fixed (I have a good hundred of links to report, although I won't be reporting them all at once, that would be so mean of me)
-The recent bandwidth limitations set up by MEGA are somewhat hindering the process, especially with the most recent uploads which are mostly hosted there. But time pays off: JD2 be blessed for queueing the files and processing everything for me overnight. 4S is still very much fine in terms of bandwidth, as long as there are enough accounts to use.
-Alternate download location added, with search feature, so you can still access the NZBs when your favourite indexer is offline. (NZBIndex and Binsearch having a 1000ish days retention, having a copy of the NZBs will come in handy. NZBKing is good but a pain to browse). On this page, you can get direct links to a NZB - which can be fed to your binaries downloader.
User avatar
Molitor
3DSL Moderator
 
Posts: 1017
Joined: Mon Feb 27, 2012 6:53 pm
Location: France
Thanks: 156
Thanked: 443 times in 324 posts

The following user would like to thank Molitor for this post
Shattered

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Scaryfun » Sat Jul 15, 2017 1:30 am

Just a note that some uploads in database might no longer be on site as they were reported down in the past but weren't deleted. So, those hundred might not actually be that much.
Image
User avatar
Scaryfun
3DSL Admin
 
Posts: 9354
Joined: Wed Jun 02, 2004 9:27 pm
Location: Toronto, Canada.
Thanks: 2142
Thanked: 921 times in 805 posts

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Shattered » Sat Jul 15, 2017 12:29 pm

Thanks again for your efforts.

I've updated my previous post with more information about the free providers. I've also changed the font colour and size to match your post.

Regarding the double or triple packing of the uploads: I think the longevity of the files is the most important.
User avatar
Shattered
3DSL Moderator
 
Posts: 2408
Joined: Tue Apr 08, 2008 10:03 pm
Thanks: 627
Thanked: 531 times in 412 posts


Return to General Topics

Who is online

Users browsing this forum: No registered users and 4 guests