Machine Learning

There's an online group called Transferred Learnings that holds monthly sessions on the latest developments. They are private though, and vet everyone to make sure you're actually working in ML.

No one does. Do you really think that engineers/researchers at OpenAI, Google Brain/DeepMind, MS Research, Meta Research etc are up to date in all topics?

We're not. We just focus on our current field of expertise/daily job for the most part. Professors at university usually have a wider (but not deeper) view, but only top ones.

Idk, depends on your standards of what it means to keep up. I skim, pick out things that seem relevant/useful to whatever I focus on right now, and put more time in that paper/blog/whatever. I think everybody does the same.

I don't - I rather focus on drilling every nook and cranny of the attention mechanism so that reading any of these papers becomes easier.

I'd say if you truly understand Transformers both theoretically and intuitively, you're already in the top 10% of MLEs. Though I'd imagine most PhDs understand it.

It's basically impossible to be completely caught up. So don't feel bad. I am not really sure it's all that useful either, you should know of technologies / techniques / architectures and what they are used for. You don't need to know the details of how they work or how to implement them from scratch. Just being aware means you know what to research when the appropriate problem comes your way.

Also a lot of the newest stuff is just hype and won't stick. If you've been in ML research since 2017 (when transformers came out) you should know that. How many different CNN architectures came out between Resnet in 2016 (or 15?) and now? and still most people simply use Resnet.

The Last Week in AI podcast is great for keeping up to date on information and you can do a deep dive on whatever catches your interest.

My tactic is to start by checking the papers that actually GET IN to major conferences (Neurips, ICLR, ICML are a good start). This narrows the search considerably. Doing a Google scholar search, for example, will just yield an insurmountable number of papers. This is, in part, due to the standard "make public before it is accepted" methodology (arXiV preprints are fantastic but they also increase the noise level dramatically).

Now, having been burnt by the chaos of the review processes of the aforementioned conferences, I am certainly aware that their publications are by no means the "Gold standard" but the notion of peer review, including the intended outcomes of improvement therethrough, is powerful nonetheless.

Honestly, I mostly just follow hugging face's blog and articles. I know there are some latest fancy attention improvements, alternatives for RLHF, GPU whatever optimization, etc, but I'm not going to implement those myself. If it's not in hugging face's ecosystem, then I most likely wouldn't use it in my daily work/production code anyway.

You don't. The process is broken, but nobody cares anymore.

1. Big names and labs want to maintain the status quo = churning paper out (and fighting on Twitter..erm X, of course).
2. If you're a Ph.D. student, you just want to get the hell out of there and hopefully try to ride a bit the wave and make some = trying to along and churn some papers out.
3. If you're a researcher in a lab, you don't really care as long as you try something that works and, eventually you have to prove in the yearly/bi/x review that you actually did some work = churn whatever paper out there.

Now, if by any chance, any absolutely crazy reason, you're someone who's actually curious about understanding the foundations of ML, deeply reason about why "ReLU" behaves like so over "ELU", or, I don't know, you question why some models with 90 billion parameters behave almost the same as a model that was compressed by a factor of 2000x and only lose 0.5% of accuracy, in brief, the science behind it all, then you're absolutely doomed.

In ML..(DL, since you mention NLP), the name of the game is improving some "metric" with an aesthetically appealing name, but not so strong underlying development (fairness, perplexity). All, of course using 8 GPU's, 90B parameters and zero replications of your experiment. Ok, let's be fair, there are some papers indeed that replicate their experiments in a total of...10..times. "The boxplot shows our median is higher, I won't comment on the variance of of it, we will leave it for future work. "

So, yes..that's the current state of affairs right there.

Try having done your PhD before SVMs were well known... yeah, the struggle is real...

Read noam shazeers work, you have now caught up

Know some general Ideas like Attention, Diffusion, Vector DB, Backprop, Dropout, Unet etc and where they work best on. Additionally know SOTA models for general use cases. When a new use case arises you should know where to dig and how to code. Have strong understanding on all these new concepts and feel free code them yourself. Most papers are just combinations of these general Ideas. If you are on a project, only then you should read in depth papers on that use case.

The main trick is learning to filter out the bs "attention aware physics informed multimodal graph centric two-stage transformer attention LLM with clip-aware positional embeddings for text-to-image-to-audio-to-image-again finetuned representation learning for dog vs cat recognition and also blockchain" papers with no code.

That still leaves you with quite a few good papers, so you need to focus down into your specific research area. There's no way you can keep caught up in all of ML.

I spend an hour each morning scanning the preprints on arxiv.org, scanning a half dozen or so and selecting perhaps one to save for weekly symposium( I like to have at least one really good paper a week to share).

It's usually easy to tell if a paper is a follow-up or response to another and if that's the case,.I might skim those too. These get supplemented with what pops up here and HN which might extend back a few months (higher signal, less noise).

This is enough to feel like I have my finger on the pulse of one topic within ML.

This website helps me:

https://hype.replicate.dev/?filter=past_day

https://www.semanticscholar.org/ helps a bit, with a bit better search than other paper repos, and some AI for recommendations... I'm sure someone will wrap a decent chat interface around their API at some point. But yea, it's bloody hard to keep up with everything happening with generative AI, especially when you include opensource work that isn't always publishing papers.

My advice is to diversify instead of specialize at this point.

​

Don't try to be the expert in NLP, branch out and learn from related fields as a way to offer insight and novelty to the noise of the constant influx of news from your area of expertise.

Perhaps we are inadvertently building a ‘Tower of Babel’.