this post was submitted on 14 Jun 2023

31 points (100.0% liked)

Technology

37712 readers

563 users here now

A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.

Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.

Subcommunities on Beehaw:

This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 2 years ago

MODERATORS

alyaza@beehaw.org

TheRtRevKaiser@beehaw.org

gyrfalcon@beehaw.org

rs5th@beehaw.org

SemioticStandard@beehaw.org

TheRtRevKaiser@kbin.social

Los@beehaw.org

coldredlight@beehaw.org

remington@beehaw.org

#lemmy /#kbin has a problem that #mastodon hasn't even attempted to solve; groups and what happens when they get popular. (mas.to)

submitted 1 year ago by schizanon@mas.to to c/technology@beehaw.org

35 comments fedilink hide all child comments

#lemmy/#kbin has a problem that #mastodon hasn't even attempted to solve; groups and what happens when they get popular.

#Communities, #groups, #magazines, whatever they are called are implemented as #Actors in #ActivityPub. They are basically just *very* popular users who boost a *lot*.

You can't just distribute them across instances the way normal actors do. Whichever server hosts @technology@lemmy.ml or @technology@beehaw.org is going to get HOSED on the regular.

top 27 comments

sorted by: hot top controversial new old

[–] sunaurus@lemm.ee 24 points 1 year ago* (last edited 1 year ago) (1 children)

While it's true that the hosts of popular communities will get more traffic, it's actually not as bad as it first seems.

Every Lemmy instance with at least one subscriber in that popular community will act as a mirror. That means that users who are just reading posts and comments will not cause any additional load on the home-instance of the popular community, because they are consuming local copies of the posts and comments.

This will actually help scaling a lot, and is in fact exactly how many centralized platforms scale (by creating a bunch of read-only copies of content).

[–] schizanon@mas.to 2 points 1 year ago (1 children)

@sunaurus what about when those users like/boost/reply?

[–] sunaurus@lemm.ee 8 points 1 year ago (1 children)

In those cases, the action will need to propagate back to the home server (that's where the "hosts of popular communities will get more traffic" comes from), but keep in mind - people usually read at least one or two orders of magnitude more than they write.

[–] schizanon@mas.to 1 points 1 year ago (1 children)

@sunaurus there's a lot of upvoting happening on popular subreddits

[–] sunaurus@lemm.ee 7 points 1 year ago* (last edited 1 year ago)

Absolutely, but a user will only upvote a post once, while they will read it on every reload of their page. (By "read" I mean "fetch it from their local mirror")

[–] sukan@feddit.jp 2 points 1 year ago (1 children)

So there is no way to horizontally scale?

[–] sunaurus@lemm.ee 5 points 1 year ago (1 children)

The network can actually scale quite well thanks to the fact that other instances will act as mirrors of communities!

[–] sukan@feddit.jp 1 points 1 year ago* (last edited 1 year ago) (2 children)

But what happens when the instance hosting the community goes down? Are all external instances still able to participate in that community? I get that they are mirrored but will everyone still be connected?

[–] Edo78@feddit.it 7 points 1 year ago (1 children)

No. The "single source of truth" is the instance hosting the community. If it goes down the community itself goes down with the ship. The only way to prevent it is to have a IT infrastructure that can provide redundancy

[–] sukan@feddit.jp 1 points 1 year ago (1 children)

Yeah, that isn't good.

[–] Edo78@feddit.it 1 points 1 year ago (1 children)

having a redundant system is feasible (I'm just a dev, not an architect so don't take my words for granted) but it have to be designed and putted together ... and prices are gonna skyrocket

[–] jmp242@sopuli.xyz 0 points 1 year ago

Lemmy / the fediverse isn't designed this way, but it could be. There are certainly systems that share diskspace and are multimaster and keep stuff as long as someone is interested in it(i.e. accessing the data). I really start to think added to the lemmy / fediverse servers should be something like what freenet used to do in terms of hosting content.

[–] BlameThePeacock@lemmy.ca 3 points 1 year ago (2 children)

If it's just a temporary outage, whatever the mirror has received prior to the outage will be shown to users on that other instance but only local interactions for that instance will update it, when it comes back up, things like votes and comments will be synchronized again across all of the instances.

For permanent outages, the community will just need to be started again on a new instance.

[–] TheAmorphous@lemmy.world 2 points 1 year ago (1 children)

But they could pick up where the now defunct community left off, right? Like, the cached copy from another server could be imported on a new server elsewhere?

[–] BlameThePeacock@lemmy.ca 2 points 1 year ago

That functionality doesn't currently exist, but migration of communities is something that's being actively talked about for development.

[–] sukan@feddit.jp 1 points 1 year ago* (last edited 1 year ago) (1 children)

[–] Whooping_Seal@sh.itjust.works 1 points 1 year ago (3 children)

Out of curiosity, how is kbin's magazine system designed to avoid this problem?

[–] sukan@feddit.jp 3 points 1 year ago* (last edited 1 year ago)

[–] sukan@feddit.jp 2 points 1 year ago

From what I understand... wait I understand nothing

[–] nii236@lemmy.jtmn.dev 0 points 1 year ago

It’s not

[–] beejjorgensen@lemmy.sdf.org 2 points 1 year ago (3 children)

What does hosed mean, technically?

[–] zero_iq@lemm.ee 6 points 1 year ago* (last edited 1 year ago)

Lots of traffic, lots of posts, lots of comments, ... That's going to need more storage, more bandwidth, more CPU power, higher running costs. The original instance hosting the community bears a higher load than the instances that duplicate it.

Ideally, there would be a way to more evenly distribute this load across instances according to their resources, but from my (currently limited) knowledge, I don't think Lemmy/ActivityPub is really geared for that kind of distributed computing, and currently I don't believe that there's a way to move subs between instances to offload them (although I believe some people may be working on that).

Perhaps the Lemmy back-end could use a distributed architecture for serving requests and storage, such that anyone could run a backend server to donate resources without necessarily hosting an instance.

For example, I currently have access to a fairly powerful spare server. I'm reluctant to host a Lemmy instance on it as I can't guarantee its availability in the long term (so any communities/user accounts would be lost when it goes down), but while it's available I'd happily donate CPU/storage/bandwidth to a Lemmy cloud, if such a thing existed.

There are pros and cons to this approach, but it might be worth considering as Lemmy grows in popularity.

[–] sukan@feddit.jp 1 points 1 year ago

It means the F word

load more comments (1 replies)

[–] codesmith@techhub.social 1 points 1 year ago

@schizanon @technology@lemmy.ml @technology@beehaw.org Yep. I expect that long-term a lot of instances will limit the ability to create groups (as Beehaw does) or place restrictions on the size a group is allowed to grow before asking that group to move to another/their own instance or ask for finances to help with the costs.

[–] Nymphioxetine@beehaw.org 0 points 1 year ago (1 children)

Did you post this from Mastodon? I wish I could tell where this came from.

Basically if I understand this right, if you have an instance with a very popular community on it. It is likely that it will need some massive infrastructure scaling if it wants to handle the enormous amount of world wide traffic?

[–] dan@upvote.au 1 points 1 year ago

I wish I could tell where this came from.

Isn't that what this colourful icon in Lemmy is for? It appears to link to the original source of the post or comment:

load more comments