I've used HTTrack in the past
It worked well for the websites that I tried it on and it's pretty easy to run. You could give it a go?
this post was submitted on 12 Jan 2024
64 points (98.5% liked)
Piracy: ꜱᴀɪʟ ᴛʜᴇ ʜɪɢʜ ꜱᴇᴀꜱ
54462 readers
318 users here now
⚓ Dedicated to the discussion of digital piracy, including ethical problems and legal advancements.
Rules • Full Version
1. Posts must be related to the discussion of digital piracy
2. Don't request invites, trade, sell, or self-promote
3. Don't request or link to specific pirated titles, including DMs
4. Don't submit low-quality posts, be entitled, or harass others
Loot, Pillage, & Plunder
📜 c/Piracy Wiki (Community Edition):
💰 Please help cover server costs.
 |
 |
|---|---|
| Ko-fi | Liberapay |
founded 1 year ago
MODERATORS
I've got a container set up of this. Drops the output on the NAS and can be accessed from any box on the local net. The only issue is it has a tendency to need the container recycled every so often, like it just gets board sitting there and quits. 🤔
You wouldn't download a website, would you?
If you were going to do somehthing like this, you might also consider doing it on some kind of version control. You might want something that does update regularly (like a wiki), but if they ever try and paywall it, you've got a copy.
Should be something you could knock out in python in an afternoon.
I’ve used wget to mirror websites. It works very well.
Wget2 can mirror websites also, and it also has the advantage of the following features that wget does not have:
- downloads multiple files in parallel, speeding things up a lot
- brotli and zstd compression support
- uses multiple proxies for parallel downloads
- supports sitemap indexes
- http2 support
This is the way
SingleFile extension can save a web page into a single html file where all media are neatly inlined inside the file. You'll have to do this manually on each page though, so it's not ideal for saving the whole website.
If you're comfortable running commands in terminal, you can use SingleFile CLI to crawl the whole website, e.g.: single-file https://www.wikipedia.org --crawl-links=true --crawl-inner-links-only=true --crawl-max-depth=1 --crawl-rewrite-rule="^(.*)\\?.*$ $1"
I use grab-site (unmaintained) for full site archival and wget -p -k for simple non-javascript single pages
I've heard good things about HTTrack, SingleFile and Archivebox but don't have any experience with them.
Archivebox looks the most modern and intuitive but is hosted on docker
grabsite works really well, can recommend
You may need a program that uses a browser (not chrome headless) to bypass cloudflare. For non-DDoS protected sites, just use any number of free open source tools.