admiralpatrick

During some work with Tess, I'd notice that my test instance was running horribly slow. The CPU was spiking, Postgres was not happy and using pretty much all the available compute.

Investigating, I found the culprit to be some crawler or possibly malicious actor sending a massive number of unscoped requests to /api/v3/comment/list. What I mean by "unscoped" is without limiting it to a post ID. I'm not sure if this is a bug in Lemmy or there's a legit use for just fetching only comments outside of a post, but I digress as that's another discussion.

After disallowing unscoped requests to the comment list endpoint (see mitigation further down), no more issue.

The kicker seemed to be that this bot / jackass was searching by "Old" and was requesting thousands of pages deep.

Requests looked like this: GET /api/v3/comment/list?limit=50&sort=Old&page=16413

Since I shutdown Dubvee officially, I'm not keeping logs as long as I used to, but I saw other page numbers in the access log, but they were all above 10,000. From the logs I have available, the requests seem to be coming from these 3 IP addresses, but I have insufficient data to confirm this is all of them (probably isn't).

• 134.19.178.167
• 213.152.162.5
• 134.19.179.211

Log Excerpt

Note that I log the query string as well as the URI. I've run a custom Nginx setup for so long, I actually don't recall if the query string is logged by default or not. If you're not logging the query string, you can still look for the 3 (known) IPs above making requests to /api/v3/comment/list and see if entries similar to these show up.

2025-09-21T14:31:59-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:00-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:01-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:01-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:12-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:13-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:13-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:13-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"

Mitigation:

First, I blocked the IPs making these requests, but they would come back from a different one. Finally, I implemented a more robust solution.

My final mitigation was to simply reject requests to /api/v3/comment/list that did not have a post ID in the query parameters. I did this by creating a dedicated location block in Nginx that is an exact match for /api/v3/comment/list and doing the checks there.

I could probably add another check to see if the page number is beyond a reasonable number, but since I'm not sure what, if any, clients utilize this, I'm content just blocking unscoped comment list requests entirely. If you have more info / better suggestion, leave it in the comments.

# Basically an and/or for has post_id or has saved_only
map $has_post_id:$has_saved_only $comment_list_invalid{
  "1:0"	1;
  "0:1" 1;
  "1:1" 1;
  default 0;
}

server {

...

location = /api/v3/comment/list {

  # You'll need the standard proxy_pass headers such as Host, etc. I load those from an include file.
  include conf.d/includes/http/server/location/proxy.conf;

  # Create a variable to hold a 0/1 state
  set $has_post_id 0;

  # If the URL query string contains 'post_id' set the variable to 1
  if ($arg_post_id) {
    set $has_post_id  1;
  }
  if ($arg_saved_only) {
    set $has_saved_only 1;
  }

  # If the comment_list_invalid map resolves to 0, "send" a 444 resposne
  # 444 is an Nginx-specific return code that immediately closes the connection 
  # and wastes no further resources on the request
  if ($comment_list_invalid = 0) {
    return 444;
  }

  # Otherwise, proxy pass to the API as normal 
  # (replace this with whatever your upstream name is for the Lemmy API
  proxy_pass "http://lemmy-be/";
}

TL;DR: Any of you who are more familiar with Fediverse platforms that aren't Lemmy/Piefed, can you let me know what the AP_IDs look like for users, posts, comments, and, if applicable, communities?

So, I've rewritten the search / search boxes in Tesseract to skip the search and directly resolve activity pub URLs for users, posts, comments, and communities. I'm loving this as it makes things so much faster and easier.

To make that work, and reduce false positives/negatives, I have to do some pre-flight checks on the URL that's submitted to the search.

Currently, it checks if the domain is to a known federated instance and looks for specific paths in the URL. If it detects the URL is an AP_ID URL, it will only resolve the object and redirect you to it (skipping the lengthy search step). For false negatives, it will pass it to the regular search but still try a federated lookup along with the search.

For Lemmy and Piefed, those are:

• /u/ for users
• /c/ for communities
• /post/ for posts
• /comment/ for comments.

For Mbin, I think it's the same except it uses /m/ for communities (they call them "magazines" I believe).

I think mastoon uses /user or maybe /username/ in the AP identifiers?

Any of you who are more familiar with Fediverse platforms that aren't Lemmy/Piefed, can you let me know what the AP_IDs look like for users, posts, comments, and, if applicable, communities?

"Antiyanks" is back at it again and has switched tactics to spamming a massive number of comments in a short period of time. In addition to being annoying (and sad and pathetic), it's having a deleterious effect on performance and drowns out any discussions happening in those posts. That spam also federates as well as the eventual removals, so it's not limited to just the posts being targeted.

Looking at the site config for the home instance of the latest ~~two~~ three alts, the rate limits were all 99999999. 🤦‍♂️

Rate limits are a bit confusing, but they mean: X number of requests per Y seconds per IP address.

The comment API endpoint has its own, dedicated bucket. I don't recall the defaults, but they're probably higher than you need unless you're catering to VPN users who would share an IP.

Assuming your server config is correctly passing the client IP via the XFF header, 20 calls to the /create_comment endpoint per minute (60 seconds) per client IP should be sufficient for most cases, though feel free to adjust to your specific requirements.

Edit: A couple of instances accidentally set the "Messages" bucket too low. That bucket is a bit of a catch-all for API endpoints that don't fit a more specific bucket. You'll want to leave that one relatively high compared to the rest. It's named "Messages" but it covers far more than just DMs.

After some back-and-forth with a few people, some very generous drinking about it, and, quite frankly, just not having a new hobby lined up yet, I've decided to un-archive the repo and continue limited development of Tesseract.

What do I mean by limited?

• By and large, it'll be in maintenance mode only.
• I'm no longer spending time here on the platform (this announcement notwithstanding). This means I will not notice annoyances and bugs like I used to. Any bug/annoyance must be user-reported in detail on Github.
• Mod tooling will likely not see any changes except maybe bugfixes. I have stepped down as mod in all communities and self-destructed my instance, so there is no longer any way for me to test privileged API calls.
• There will probably never be support for Lemmy 1.0 or anything beyond the current 0.19.x API
• Piefed support is still up in the air; if/when Piefed support happens, it will be in place of Lemmy and not concurrent with.
• The Matrix rooms ~~and support/announcement community~~ are not coming back. All bugs, questions, etc will need to be submitted via Github. New versions will not be announced other than in the "Releases" list on Github
• ~~The unlocked, hosted instance formerly at "tesseract.dubvee.org" will not be coming back; you will need to self-host or ask your instance admin(s) to offer it as an alternative UI.~~
• I'm more likely to remove features and fine tune what's left than add anything new.
• I am taking a more opinionated approach to options, settings, etc. The codebase has become a fustercluck, so some options are likely on the chopping block in favor of "this is just how it is"

1.4.42 is in development.

NBC New York obtained exclusive video that shows the torture victim running down the street after his escape, 17 days into captivity, and seeking help from a police officer.

What to Know

• An Italian man escaped from a house on Prince Street in New York City last week, where he said he had been held captive for 17 days and allegedly tortured by two business partners
• Two NYPD members, including a detective on Mayor Eric Adams' security detail have been placed on modified desk duty after links surfaced to the two crypto businessmen charged with kidnapping an Italian tourist, sources tell News 4
• The detective allegedly provided security for the Prince Street townhouse where the Italian man was held and it's believed he may have picked up the tourist from the airport and brought him to SoHo, sources tell News 4
• City Hall said it is "disturbed by these allegations" and the investigation into the officer's behavior is ongoing
• John Woeltz, was arrested in his bathrobe outside the scene; he is expected in court on kidnap and torture charges, among others, on Wednesday. A second suspect, William Duplessie, surrendered in connection with the case a day ago

Note: There are two headlines for this, one on the actual article page and a different one in the embed description. The post title is the one from the article. The headline from the embed description is "Crypto king torture investigation takes shocking turn".

“I would not acknowledge reproduction as a human right, but instead as a form of rape,” IndictEvolution wrote on Lemmy.World in July 2023. “I am also not bothered by infanticide as long as it is done humanely...”

Preface: I'm neither equipped nor here to diagnose anyone with body dysphoria or anything like that.

I totally get the appeal of working out, getting a nice summer/beach body, staying fit/healthy and all that. That's all well and good. But the degree to which bodybuilders intentionally overdo it just looks awful to me. Like, to me, they all look like tiny little heads atop roided-out, spray-tanned, lumpy, disproportionate looking bodies.

That just looks gross to me, and I can't see the appeal of wanting to do that to yourself.

Just added rule 6 to the sidebar that reduces some ambiguity between rules 4 and 5. 99% of posts here already do this, so there shouldn't be much change other than it being required now.

Rule 6: Defend your position

This is a bit of a mix of rules 4 and 5 to help foster higher quality posts. You are expected to defend your unpopular opinion in the post body. We don't expect a whole manifesto (please, no manifestos), but you should at least provide some details as to why you hold the position you do.

This won't be applied retroactively, but anything from here on out is expected to include some exposition to go along with the opinion itself

Edit 2025-01-13: LW has indicated they will be clarifying these rules soon. In the mean time, the community will remain locked until those are updated and deemed acceptable.

So the LW Team put out an announcement on new, site-wide moderation policy (see post link). I've defended, to many a downvote, pretty much every major decision they've made, but I absolutely cannot defend this one. In short, mods are expected to counter pretty much every batshit claim rather than mod it as misinformation, trolling, attack on groups, etc.

My rebuttal (using my main account) to the announcement: https://dubvee.org/comment/3541322

We're going to allow some "flat earth" comments. We're going to force some moderators to accept some "flat earth" comments. The point of this is that you should be able to counter those comments with words, and not need moderation/admin tools to do so.

(emphases mine)

Me: What if, to use the recent example from Meta, someone comes into a LGBT+ community and says they think being gay is a mental illness and /or link some quack study? Is that an attack on a group or is it "respectful dissent"?

LW: A lot of attacks like that are common and worth refuting once in awhile anyway. It can be valuable to show the response on occasion

I understand what they're trying to address here (highly encourage you to read the linked post), but the way they're going about it is heavy handed and reeks of "both sides"-ing every community, removing agency from the community moderators who work like hell to keep these spaces safe and civil, and opening the floodgates for misinformation and "civil" hate speech. How this new policy fits with their Terms of Service is completely lost to me.

I'll leave the speculation as to whether Musk dropped LW a big check as an exercise to the reader.

For now, this community is going dark in protest and I encourage other communities who may disagree with this new policy to join. Again, I understand the problem that is trying to be addressed, but this new policy, as-written, is not the way to do it.

Voting has concluded on whether the community wants to remove the "Vote the opposite of the norm" voting guidelines.

As you can see in the screenshot below (or in the post), the results are a tie (only upvotes are counted, not the score). I abstained from the vote, leaving it entirely to the community, and I do not wish to cast the tie-breaking vote.

Since it is a tie, I'll treat that as a non-majority vote and, as such, we will keep the voting guidelines as they are.