XTJ7

Thanks for clarifying that :)

The client is not, no. Wireguard is open source and you can selfhost headscale, which is an open source server for tailscale, provided by tailscale themselves.

To be clear: I too am using tailscale for its convenience and reliability. While I havent had any issues with wireguard clients, it is interesting to see that there may be cases where switching from wireguard to tailscale can actually still make sense.

Oh, that is crazy! I think I should do a bit of performance testing then :)

Not really, no. Tailscale uses wireguard under the hood. It has a nice user interface and makes setting up a split VPN super easy. It also provides relatively easy ways to do ACL between devices. If you already got wireguard set up, you can skip tailscale.

Also an i7 of that age doesn't go up to 12 cores, for multithreading that is quite a benefit. Especially in a server.

For sure! But the M1 ultra still holds up really well. I doubt I will replace it for another 3 years at the very least. Currently CPUs are progressing at an impressive rate across the board. Would I like an M3 ultra? Sure, but do I really need it? Sadly no :) The upgrade to an M5 ultra will be insane though.

In my case it's an Epyc 7642 with 8x64GB DDR4 2666, so that may be why my generation is significantly slower.

I find anything below 5 tokens per second not really usable, so that's why I stick with my M1 Ultra. It has plenty of really fast RAM and that again explains most likely why it performs so well, if LLMs are that dependend on fast memory.

I also have a 3090 in another machine but that's also just 24gb and I don't want to shell out more money right now for playing with LLMs, if the M1 Ultra is doing good enough :)

I tried a 70B model on my 48 core epyc with 512gb RAM and it was unusable. I think 1.5t/s or so? Even if you double that it's not great. My M1 Ultra runs it comfortably at 6-7t/s and sips power.

Probably a dual 3090 setup would be the most cost effective solution at the moment while the M1/M2 ultra are the most power efficient solution.

I am also strongly considering to make the switch now. Any notable drawbacks/learnings from using it for 6 months?

Had a business partner with whom I met up face to face regularly. He was supposed to do the sales and help getting a market fit due to his experience and contacts in the industry.

I designed and built the app, set up the server and infrastructure, bought the domains etc. But despite his father owning a large company in that industry, he couldn't even get them to use it for free. Not because the app didnt work, we tested it and everything worked fine. It was even in use at his fathers company for a short while as we temporarily had a customer relationship manager who pushed for it. Once she left, it all fell apart and my partner did basically nothing but talk out of his ass.

I ended up scrapping the thing and focusing on an industry where I have experience and contacts, but the lesson is: face to face contact doesn't do shit if your partner isn't willing to get the job done (he seems to have been excited at creating something but far less excited with actually running it).