Programming

I would separate concerns. For the scraping, I would dump data as json onto disk. I would consider the folder structure I put them into, whether as individual files, or a JSON document per line in bigger files for grouping. If the website has good URL structure, the path could be useful for speaking author and or id identifiers in folders or files.

Storing json as text is simple. Depending on the amount, storing plain text is wasteful, and simple text compression could significantly reduce storage size. For text-only stories it's unlikely to become significant though, and not compressing makes the scraping process, and potentially validating completeness of scraped data simpler.

I would then keep this data separate from any modifications or prototyping I would do regarding modification or extension of data and presentation/interfacing.

Definitely SQLite. Easily accessible from Python, very fast, universally supported, no complicated setup, and everything is stored in a single file.

It even has a number of good GUI frontends. There's really no reason to look any further for a project like this.

SQL is designed for querying (it's a query language lol). If the stories are huge, you can save them to individual files and store the filepath in the database, but otherwise it can hold columns with a fair amount of data if needed.

You can probably get away with using sqlite. A more traditional database would be postgres, but it sounds like you just need the database available locally.

Put them into an opensearch database. It is the open source fork of elasticsearch. It has an sql plugin so you can retrieve the raw data the usual way. And there is probably also an integeation/library for it if you use any major framework/language in the backend.

But on top of it you get a very performant full text search. This might come in handy for example when you remember a sentence from a story, or if you want to find all stories with a specific character name or word for whatever reason.

I would scrape them into individual json files with more info than you think you need, just for the sake of simplicity. Once you have them all, then you can work out an ideal storage solution, probably some kind of SQL DB. Once that is done, you could turn the json files into a .tar.zst and archive it, or just delete them if you are confident in processed representation.

Source: I completed a similar but much larger story site archive and found this to be the easiest way.

Python sqlite3 module for the metadata and it has some features now for full text search that can probably handle a few thousand stories. For a bigger collection like ao3, try solr.apache.org or elastic search etc.

Based on my experience...

• JSON is good for storing complex but readable data.
• SQL is used case for demanding online loads. (Not exactly ideal for serialization)
• YAML is a strange beast, but I've mostly seen it used for configuration.

If scraping is reliable, I'd use the classic python pickle or JSON.dump

For a few thousand I would just use a sqlite dB...

3 tables:

• Story with fields: Id, title, text
• Meta with fields: Id, story-id, subject, contents
• Tags with fields Id, story-id, tag

Use SQL joins for sorting etc.

Sqlite is easily converted to other formats if you decide to use more complex solutions.

Since you want to be able to access these stories as well as store them, you can kill two birds with one stone by creating a Django app with a SQLite backend. The builtin admin site will let you browse and search the content without having to write much code.

I do a lot of web services and I'm a big fan of SQL, but I wouldn't use a SQL database for this myself. Something like MongoDb or Cassandra would probably serve you better (depending on whether you prefer a REST interface to your data or something more conventional). You've got a very flat structure except for tags.

Tags are the one feature that might make me choose SQL due to the many: many relationship.

I'm not sure what role you think YAML would play. You could store each story as YAML, but then you'd have to parse basically everything to filter and sort. The story should just be a massive text field, and the metadata goes into respective fields. Tags might be comma delimited or in SQL you could normalize it so that you have three tables: stories, tags, and a table that basically looks like

I'd at least first try to use a non-relational database structure because filtering and sorting by tag might still be fast enough. If it's too slow then you could go SQL, but I'd aim for the less complex solution.