Index ⇒ General Topics ⇒ Operation LW2Usenet - Backing up our uploads on newsgroups

Thoughts on life, the universe and everything else not covered in other categories.

Moderator: LW Moderator

Operation LW2Usenet - Backing up our uploads on newsgroups

Postby Molitor » Wed May 31, 2017 8:43 pm

TL;DR
LW uploads are being uploaded to newsgroups for backup purposes.
Click HERE and HERE for an introduction to Usenet.
Click HERE for two free providers.
Click HERE to find the NZBs.


Introduction & background
In a 2016 post, Shattered outlined the possibility of using newsgroups to back up our uploads. His test upload from September 2010 can be found here for reference.
The outstanding benefit of newsgroups is their retention, meaning how long the files remain available on the servers. Unlike cyberlockers, files posted on newsgroups do not "expire" depending on their popularity; and nowadays, many providers reach 3000+ days of retention, meaning you can access files posted 8 years ago --provided they haven't been DMCA'd meanwhile-- and that files uploaded now should remain available for the next 8 years --provided they aren't DMCA'd in the meantime--. Binary retention is still growing for most of the providers, on a daily basis.
What I'm looking forward to is to provide a mirror for all LW uploads - from the oldass early 2000s Rips to the latest ISOs.


The main downsides
Because it can't be perfect.
  • First major disappointment: Subscribing for a good Usenet access is not free. However it is usually very cheap. If you are a humble downloader, looking for a few MBs there and there, a free trial at a major provider, or the free plan at Free-Usenet (1 Mbit/s = ~140 KB/s) will leave you satisfied. If you need to leech some terabytes off newsgroups, a paying subscription is probably what you'll need.
  • Second drawback: all files meant to be backed up for LW2Usenet are repacked and passworded before they are posted to newsgroups. This implies that double-packed scene releases, for example, become triple-packed, hence you will sometimes get files insides archives inside archives inside archives, à la matrioshka. Lately I've been trying to avoid this by removing the first layer of compression before processing the files, however the practice is still subject to debate, I would like to hear your opinion on it.
    This also means that in order to get a specific file inside these encrypted archives, you need to download the whole set of archives. E.g, you want rld-btaa.r60 ? It's only 95MB, but you'll need to download the whole 8GB set and decompress it to do so. Which brings us to the third part:

What's the point then?
Providing a long-term, last-chance backup for uploaders, and a viable option for everyday downloaders.
In other words, if you are an uploader, this mirror will likely only be useful for you in case you loose files and wish to reupload them to a cyberlocker.
However, if you are a downloader, chances are that you chose to download off Usenet because the original upload was broken in some way - remember you will need to download the whole game.


Modus operandi
Files downloaded off LW will be packed into 200MB archives and uploaded to alt.binaries.archive.encrypted - with 10% recovery volume.
If the original download is broken or misses files, I upload a replacement instead when possible, e.g when it's a scene release or something easy to find - otherwise all I can do is skip it, report it and process it later if a reupload happens.
There are two movements to this leeching process. The first part consists in mirroring the uploads #1 to #9999. In the meantime, I'm also processing uploads starting from #10000 in the hopes of catching up the latest uploads (#14100 and so on as I'm writing this).


Stats
Image
User avatar
Molitor
3DSL Moderator
 
Posts: 1064
Joined: Mon Feb 27, 2012 6:53 pm
Location: France
Thanks: 164
Thanked: 467 times in 342 posts

The following user would like to thank Molitor for this post
Scaryfun, Shattered, user43687

Operation LW2Usenet - Backing up our uploads on newsgroups

Postby Molitor » Wed May 31, 2017 9:11 pm

Where to find them
For the uploads that have been already processed, you may find them on your favourite indexer:
Code: Select all
https://nzbindex.com/search/?poster=LW2Usenet+%3CLW2U%40Legendsworld.net%3E&more=1
https://binsearch.info/?server=2&max=250&g=alt.binaries.archive.encrypted&a=LW2Usenet+%3CLW2U%40Legendsworld.net%3E
http://www.nzbking.com/poster/LW2Usenet%20%3CLW2U%40Legendsworld.net%3E/

Alternate download location (with search feature!) (Randomly on and offline, depending on my mood)
Code: Select all
http://163.172.58.206:8080/LW2U/


The file sets are named after their LW upload number, and presented as such: LW_[mirror n°].
Ex: You want to download "Batman: Vengeance - AlcoholClone ISO". Notice in the URL how the mirror number is specified: "download.php?mirror=10100". So this upload is mirror #10100. Thus it can be found as "LW_10100" on your indexer.

All archives are protected with the password
LW
Keep in mind that even after unpacking these files with the LW password, the files inside may require a second password, provided in the Uploader's notes.
User avatar
Molitor
3DSL Moderator
 
Posts: 1064
Joined: Mon Feb 27, 2012 6:53 pm
Location: France
Thanks: 164
Thanked: 467 times in 342 posts

The following user would like to thank Molitor for this post
Shattered

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Scaryfun » Thu Jun 01, 2017 2:57 am

That's a massive undertaking. :thanks: for all your efforts. I'm doing an update tonight, but I'll check tomorrow. It's sad when we lose games that have been upped before. So, with this, it will prevent further losses.
Image
User avatar
Scaryfun
3DSL Admin
 
Posts: 9475
Joined: Wed Jun 02, 2004 9:27 pm
Location: Toronto, Canada.
Thanks: 2258
Thanked: 959 times in 839 posts

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Shattered » Thu Jun 01, 2017 6:25 pm

Great job!
User avatar
Shattered
3DSL Moderator
 
Posts: 2422
Joined: Tue Apr 08, 2008 10:03 pm
Thanks: 636
Thanked: 543 times in 422 posts

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Shattered » Sun Jul 02, 2017 11:06 am

XS Usenet has raised the retention to 1100 days and speed to 2 Mbit for free accounts, but with a 25 GB data limit (not sure if this is per month). Combined with a Free-Usenet account you can download posts up to 3 years old with a reasonable 350 kB/s speed without paying a cent.

XS Usenet

- Download speed: 2 Mbit/s
- Connections: 5
- 25 GB data limit
- Unlimited days
- No SSL
- 1100 days binary retention


Code: Select all
https://www.xsusenet.com/


Free-Usenet

- Download speed: 1 Mbit/s
- Connections: 1
- Unlimited traffic
- Unlimited days
- Free SSL included
- 2700+ days binary retention

Note:

A free account needs to be renewed every 24 hours, and the password changes with every renewal.


Code: Select all
https://free-usenet.com/
User avatar
Shattered
3DSL Moderator
 
Posts: 2422
Joined: Tue Apr 08, 2008 10:03 pm
Thanks: 636
Thanked: 543 times in 422 posts

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Molitor » Fri Jul 14, 2017 5:59 pm

:idea: Update on the progress after approx. 1 month 1/2 of running the service:
-Uploads #1-1500 are now covered, as well as #10000-10600.
-Many have been skipped, though: most of them are rips uploaded by the long-away-now MasteromaN. All of these rips have an ISO equivalent anyway, so they are not exactly high-priority stuff. There's a bit of backlog due to incomplete downloads awaiting to be fixed (I have a good hundred of links to report, although I won't be reporting them all at once, that would be so mean of me)
-The recent bandwidth limitations set up by MEGA are somewhat hindering the process, especially with the most recent uploads which are mostly hosted there. But time pays off: JD2 be blessed for queueing the files and processing everything for me overnight. 4S is still very much fine in terms of bandwidth, as long as there are enough accounts to use.
-Alternate download location added, with search feature, so you can still access the NZBs when your favourite indexer is offline. (NZBIndex and Binsearch having a 1000ish days retention, having a copy of the NZBs will come in handy. NZBKing is good but a pain to browse). On this page, you can get direct links to a NZB - which can be fed to your binaries downloader.
User avatar
Molitor
3DSL Moderator
 
Posts: 1064
Joined: Mon Feb 27, 2012 6:53 pm
Location: France
Thanks: 164
Thanked: 467 times in 342 posts

The following user would like to thank Molitor for this post
Shattered

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Scaryfun » Sat Jul 15, 2017 1:30 am

Just a note that some uploads in database might no longer be on site as they were reported down in the past but weren't deleted. So, those hundred might not actually be that much.
Image
User avatar
Scaryfun
3DSL Admin
 
Posts: 9475
Joined: Wed Jun 02, 2004 9:27 pm
Location: Toronto, Canada.
Thanks: 2258
Thanked: 959 times in 839 posts

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Shattered » Sat Jul 15, 2017 12:29 pm

Thanks again for your efforts.

I've updated my previous post with more information about the free providers. I've also changed the font colour and size to match your post.

Regarding the double or triple packing of the uploads: I think the longevity of the files is the most important.
User avatar
Shattered
3DSL Moderator
 
Posts: 2422
Joined: Tue Apr 08, 2008 10:03 pm
Thanks: 636
Thanked: 543 times in 422 posts

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Molitor » Thu Jan 11, 2018 5:35 pm

Stats are available now, mainly for self-gratification purposes but also to satisfy potential curiosity about the ongoing progress. A few notes about them:
-The success rate indicates how many uploads have been successfully backed up compared to the overall number of uploads in the range. For instance the 2000-2999 range theoretically contains 1000 uploads, but it's not the case: as mentioned above by Scaryfun, some old uploads are still present in the DB while they are not linked anymore to any game entry. There is also a number of "blank" entries which do not contain any link.
-Range 1-999 has a lower success rate because most of its uploads are old were broken long before I started backing them up. But most of them are rips which already have an ISO equivalent available.
-For the aforementioned reasons, a lower success rate doesn't mean that many important uploads have been skipped, and a 100% success rate is unachievable.
-At some point I will try to correct the stats by removing the blank and unused mirrors from the overall number of uploads. I could do so with a script that check every mirror page and looks up for the words "No links found". But even with an adjusted number, the success rate wouldn't change that much. Out of 5200 mirrors (#1 to #3400 + #10000 to #11800) I have checked so far, there are ~370 which have not been processed but are fixable and maybe ~200 which are unused and thus will never be processed.
-I also note down the size if each backup range, so we can have a slight idea of the volume of data involved.
User avatar
Molitor
3DSL Moderator
 
Posts: 1064
Joined: Mon Feb 27, 2012 6:53 pm
Location: France
Thanks: 164
Thanked: 467 times in 342 posts

The following user would like to thank Molitor for this post
Scaryfun, Shattered

Re: Operation LW2Usenet - Backing up our uploads on newsgrou

Postby Scaryfun » Fri Jan 12, 2018 9:15 am

That's amazing work, Molitor! :D
It's also valuable to know which ones you find that are not working currently , that way two birds are killed with one stone.
We should then put links directly to usenet from each game entry, so people know to use it.
Image
User avatar
Scaryfun
3DSL Admin
 
Posts: 9475
Joined: Wed Jun 02, 2004 9:27 pm
Location: Toronto, Canada.
Thanks: 2258
Thanked: 959 times in 839 posts


Return to General Topics

Who is online

Users browsing this forum: No registered users and 1 guest