1092
this post was submitted on 05 Oct 2023
1092 points (98.1% liked)
Technology
59402 readers
2532 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Surprised to see so many plugging kagi in this thread. A subscription to search the internet seems crazy to me. Is it that good?
This article is a pretty good summary of why, by Google's own words, an ad driven search experience will be rubbish:
https://pluralistic.net/2023/10/03/not-feeling-lucky/#fundamental-laws-of-economics
Not only does Kagi produce great search results, as good as "old Google" IMO, its business model means the above cannot (or at least, shouldn't) happen. If it ever changed its model to include ads etc it would collapse so fast.
So for me, unlike the other poster, I'd recommend it to everyone who's finding the existing search engines are rubbish and full of useless Etsy and SEO etc links.
I can't find any information about their search engine crawler. Isn't it standard for search engines to label their crawlers or something?
It’s conventional to do this, but a user agent string is entirely up to the client, and robots.txt is just a suggestion.
So for the best results, you probably want to mock Google’s crawler because it’s suicidal to block that if you want search traffic.
https://help.kagi.com/kagi/search-details/search-sources.html