marsara9

joined 1 year ago
[–] marsara9@lemmy.world 3 points 6 months ago

Yes it would. In my case though I know all of the users that should have remote access snd I'm more concerned about unauthorized access than ease of use.

If I wanted to host a website for the general public to use though, I'd buy a VPS and host it there. Then use SSH with private key authentication for remote management. This way, again, if someone hacks that server they can't get access to my home lan.

[–] marsara9@lemmy.world 6 points 6 months ago (2 children)

Their setup sounds similar to mine. But no, only a single service is exposed to the internet: wireguard.

The idea is that you can have any number of servers running on your lan, etc... but in order to access them remotely you first need to VPN into your home network. This way the only thing you need to worry about security wise is wireguard. If there's a security hole / vulnerability in one of the services you're running on your network or in nginx, etc... attackers would still need to get past wireguard first before they could access your network.

But here is exactly what I've done:

  1. Bought a domain so that I don't have to remember my IP address.
  2. Setup DDNS so that the A record for my domain always points to my home ip.
  3. Run a wireguard server on my lan.
  4. Port forwarded the wireguard port to the wireguard server.
  5. Created client configs for all remote devices that should have access to my lan.

Now I can just turn on my phone's VPN whenever I need to access any one of the services that would normally only be accessible from home.

P.s. there's additional steps I did to ensure that the masquerade of the VPN was disabled, that all VPN clients use my pihole, and that I can still get decent internet speeds while on the VPN. But that's slightly beyond the original ask here.

[–] marsara9@lemmy.world 8 points 9 months ago* (last edited 9 months ago) (1 children)

A couple of options in my opinion, as I just did this myself:

You can use the CLI tool to "upload" them. You can even do this from the server itself. So upload times would be as fast as your network card can process or however fast your server is, whichever is slower. It does require that you create an API key for the user in question though.

Otherwise you can create an external library and link that to your account. Now Immich will still index this library but it won't move or manage the actual files. I'm not sure though if it looks at those files for duplicates (i.e. if you try and upload the same photo from your phone to the server). This external library will also prevent deleting photos as well, FYI.

There might be other options that I'm not aware of, as I've only been using Immich for about a month now.

Edit: link to the CLI documentation: https://immich.app/docs/features/command-line-interface/

[–] marsara9@lemmy.world 2 points 1 year ago

Correct. As I can only provide links to posts that are on your selected home instance. Eventually I'll change this but you'll get a 404 page for links that aren't on your home instance, but see my P.S. below.

P.s. there have been changes to the Lemmy API that have prevented me from getting updates for about a month now. So most of the results you're seeing are from old posts only. Until I can rebuild the crawler or find a new API there won't be any new content.

[–] marsara9@lemmy.world 5 points 1 year ago* (last edited 1 year ago)

This is the same reason I had to turn off my search engines crawler.

There were changes made to the API to ignore any page > 99. So if you ask for page 100 or page 1_000_000_000 you get the first page again. This would cause my crawler to never end in fetching "new" posts.

lemm.ee on the other hand made a similar change but anything over 99 returns an empty response. lemm.ee also flat out ignores sort=Old, always returning an empty array.

Both of these servers did it for I assume the same reason. Using a high page number significantly increases the response time. It used to be (before they blocked pages over 99) that responses could take over 8-10 seconds! But asking for a low page number would return in 300ms or less. So because it's a lot harder to optimize the existing queries, and maybe not possible, for now the problematic APIs were just disabled.

[–] marsara9@lemmy.world 5 points 1 year ago

I was using it's APIs. But new restrictions have effectively been put in place that prevent me from using them for what I need. Similar API calls were being made that were causing DDOS attacks on lemmy.world.

As for running a lemmy instance itself. That's a thought but I need the data in a different format to do efficient searches. It's a tricky problem.

[–] marsara9@lemmy.world 6 points 1 year ago (3 children)

Yep that's the new idea. The sad part is that with this method there's no way to get historical data. Only new posts. So if a server goes down, gets DDOSd etc... I'll lose posts forever.

Also building an ActivityPub implementation from scratch isn't trivial either. So that'll take some time.

I've got a few other ideas I'm playing with as well. Like just assuming that internal post IDs are all sequential and literally fetching them one by one. Or maybe some combination of both?

[–] marsara9@lemmy.world 38 points 1 year ago (6 children)

Thanks for the shout-out.

But FYI I've run into some bugs that's preventing new content from being indexed. So you won't see anything new (from about a week ago) until I can find a new method to fetch new posts.

[–] marsara9@lemmy.world 2 points 1 year ago

Heads up your copied post just went to the same community, so it looks like you have duplicate postings.

[–] marsara9@lemmy.world 2 points 1 year ago* (last edited 1 year ago) (6 children)

Ya if it's not federated already, I'm thinking the link will just remain as-is? Then comes the question of what happens after federation is complete? Ideally I'm thinking to do these rewrites as the comment containing the link gets federated so it only happens once, but that prevents updates... If I do it during someone looking at the post, then that becomes a lot of network traffic...

Still a lot to plan out initially and then propose to the devs for feedback... So it'll be awhile still.

[–] marsara9@lemmy.world 8 points 1 year ago (8 children)

I just got a PR merged today that might help with this. I'll start experimenting with it more over the next week or so.

Basic theory is that I can detect at least lemmy posts in the comment bodies and then rewrite those to your local instance. Primarily question is going to be performance, as remote network calls will be necessary.

[–] marsara9@lemmy.world 5 points 1 year ago

You can always try constructing the URL manually. I know this sucks, but it might be a workaround.

https://lemmy.ml/c/songaweek@lemmy.world

Long story short you just have to go to lemmy.ml and then add /c/ followed by the community name @ the instance it belongs to.

 

I keep see people complaining about not being able to find active communities that match their interests. So I've added a new feature to https://www.search-lemmy.com/ that allows you to search posts for a particular topic and then it tells you which communities have the most posts matching your search query.

And assuming that you've set your home instance correctly, those links will even open up in your home instance, so that you can subscribe directly to them.

For example, if you search for 'linux' (https://www.search-lemmy.com/find-communities/results?query=linux&page=1) it gives you a link to each community, tells you which instance it's on and how many matches it found for your query.

All of the same filters that you can use on the normal search can be used here as well. So if you just want to find the best community that mentions linux on lemmy.world (https://www.search-lemmy.com/find-communities/results?query=linux+instance%3Alemmy.world&page=1), you can filter by just that instance. Click on the Search Tips button to see a list of all of the available filters.

P.S. I'm aware of https://lemmyverse.net/ etc... and while those are great as well, this allows you to search to see what people are actually talking about on the various communities.

Again, if you have any feature requests or find any bugs, PLEASE reach out or ideally go to my github (https://github.com/marsara9/lemmy-search) and log a bug there.

 

A couple days ago I updated https://search-lemmy.com/ to 0.4.0.

New features, that several people were asking for:

  • The UI has been overhauled and it should be much easier to find your home instance now.
  • Search itself has been overhauled. Increase search performance significantly. I also automatically search for related terms as well. You may now see fewer search results, but ideally they should be more relevant. You can also now include basic syntax like:
    • quotes: "some terms that must be together"
    • negative terms: cat -dog (shows posts about cats that don't mention dogs)
    • either or: cat OR dog (shows posts about either cats or dogs). The default search behavior is now an implicit AND, but order doesn't matter.
  • I've added several new filters that you can use including:
    • !safeoff -- Disables safe search allowing NSFW posts to appear in the search results (NSFW is now hidden by default)
    • since:YYYY-MM-DD -- shows only posts that have occurred since the specified date
    • until:YYYY-MM-DD -- same as above but in reverse. It will only posts up to the given date.
  • I've removed the preferred-instance query parameter from the results URL so it should be easier to share links to search results now.
  • The date the post was created or last updated is now displayed in the search results.

Bug Fixes:

  • Site performance should now be stable. Fixed a bug related to the database pool that was causing the site to hang.
  • Fixed a bug that would cause broken links.
  • Fixed various bugs with the crawler causing posts to be missed.

Known Issues:

  • If you set your home-instance to a fairly small instance, the number of search results is also relatively small. Once (https://github.com/LemmyNet/lemmy/issues/3259) is resolved. I should be able to show links regardless of what your home instance is set to, allowing you to search the entire Fediverse.
  • Currently searching only looks at the post title and body. Comments aren't indexed either. This also is dependent on the above issue on Lemmy itself.

Finally some things to note:

I've started to refactor the code to abstract away Lemmy from the actual search engine. As I now start to prepare to search other Fediverse instances like Kbin, and maybe even Mastodon, etc...

37
submitted 1 year ago* (last edited 1 year ago) by marsara9@lemmy.world to c/fediverse@lemmy.world
 

I shared bits and pieces of this before, but it's officially up and running now: https://www.search-lemmy.com/

This is an enhanced search engine for Lemmy. With a few primary goals:

  • You can choose a preferred instance. After choosing what your primary instance is, and performing a search ALL links will open in that instance.
  • This aims to be a replacement for using site:reddit.com in Google, but just for the fediverse.
  • You can filter the search results by:
    • Instance -- This will filter the results to only show communities that belong to a particular instance. Just type something like instance:lemmy.wrold or instance:https://lemmy.world/. This is separate from your preferred instance, such that you can search for posts on lemmy.world while still opening them on lemmy.ml.
    • Community -- You can refine the search by a specific community. You use the same syntax that you'd use here community:[!fediverse@lemmy.world](/c/fediverse@lemmy.world).
    • Author -- Similar to the above you can also filter by a specific author such as: author:@marsara9@lemmy.world.
  • The entire thing is open-source. You can view the code and even host your own instance... See more details here: https://github.com/marsara9/lemmy-search.

NOTE: This only supports Lemmy instances for now. Other fediverse type instances may be in the future depending on how this works out.

I've been working on this over just the last few weeks, so it hasn't had a chance to crawl much of the fediverse yet. For now it only supports lemmy.world and lemmy.ml but other preferred-instances will come online as time goes by.

If anyone finds any bugs, and I'm sure you will, or if anyone has any suggestions PLEASE raise an issue on GitHub for me to track. Lastly, if anyone wants to help contribute please feel free to reach out.

NOTE TO SERVER ADMINS: You can prevent your site from being crawled by adding lemmy-search to your robots.txt for the user-agent.

view more: next ›