Machine Learning

Hi I am a PhD student and work with PINNs. Technically my department is CS but I have a physics background and work in a group of physicists. I agree with your thoughts that most papers show a toy example and call it a day. I feel that more rigour is definitely needed for PINNs to be useful as replacements for numerical methods. Right now PINNs are not used for big scientific problems precisely because there are no error bounds or theoretical guarantees. Training is very finicky as well. It still seems to be early days, and the field might become more useful as it matures. Somebody with a math background and more mathematical rigour would be a welcome addition!

Finding the research gap is part of the phd process. It can be demotivating but think of your phd as an apprenticeship. You shouldn't expect grand results while you are learning to be a scientist. If there is a half-assed paper, you can still cite it and continue working on your own thing and publish it.

I would be happy to discuss more, not sure what I can add.

Your observations are spot-on. PINNs are indeed mostly a terrible way to solve a differential equation, and I would definitely recommend doing a PhD on something else :)

However, the broader field of sciML itself is booming right now -- this is an excellent choice to work in! Let me give you some examples.

• On the bio side there is a lot of work happening in applying LLMs/diffusions to modelling proteins, for drug discovery, ...
• Neural differential equations are a great way of improving existing diffeq models of many phenomena (chemical kinetics, population dynamics, weather modelling, ...).
• I've seen a lot of use-cases for ML in astrophysics recently -- describing priors for black hole imaging, a fair bit of symbolic regression, ...
• On the computational side there have been great recent strides in building an "autodifferentiable everything" software stack, e.g. a lot of my recent work has been about building "scipy in JAX".
• This has actually brought with it some new numerical methods too; e.g. I've been involved in developing some new numerical SDE solvers, and some new quasi-Newton optimisers.

And so on and so on. You'll find a similar story across most of science these days; I'm throwing out some of the examples above that I'm most familiar with.

You also touch on the difficulty of coming up with ideas right now. First of all, don't stress about that -- this is totally normal in your current position. I strongly believe ideation stems primarily from a technical depth of knowledge, and that's something you only really develop a couple of years into your PhD. (And after that, trust me, you'll have the opposite problem: too many ideas and not enough time to pursue them all.)

Find a good advisor and they'll be more than happy to hand you off some of the ideas they haven't had time to tackle themselves; this is the fairly standard arrangement for most PhD students and their supervisors.

You might like Just Know Stuff, which a post I've written on topics worth knowing in ML, and How to handle a hands-off supervisor, which also includes a lot of advice on tackling a PhD more generally.

For context, my own background was mathematics originally. I've since wandered through CS, machine learning, and now focus primarily on problems in numerics and in computational biology.

I am finishing phd in next few months, from math undergrad. i spend a lot of time on my own, don't like heaps of stimulation, like to think things through properly to understand them. hence study math. i hate the currnt situation with keeping up with papers, mainly because it feels random and very noisy, but i've learned over time that it's easy to manage, a skill u develop. ok so right now with papers etc the community seems to be changing - as you said it's way too hard to keep up with everything. journals do not work as well as aggregators (as they used to) for relevant novel contributions for a few reasons. obviously increased ML reseaaaach is the main one. but its like - submissions to cvpr for this round are like 20,000. i think last year was 17,000? idk exact. but - this growth is insane. my supervisor said maybe 5 years ago you would be lucky to get 2,000 submissions. which worked well, they could find enough reviewers who were knowledgeable enough to accept or reject the paper. and cvpr was the goat for my supverisor's area. but now, because of the high submissions, the extra 18,000 reviewers are often less experienced and feedback / outcomes are more random from the author's perspective. some great work coming out obviously. but. ppl still need to publish, so paper tanks emerge to reduce probability of no publications for current round, less effort goes into any single paper, cos u have to strategise. what would a probability theorist do. etc. on top of this, there are now more authors not from academia - industry researchers eg google snap or also startups benefit greatly from having a submission in eg neurips, good to show investors that u have actually done something, so it's like advertising also now. so right now, journals do not serve as effective aggregators of information. i think this was their original motivation. i think over time we will move towards community ranked publications eg sanity arxiv from karpathy ranks relevant publications using popularity, maybe this could augment using cited by numbers. even still, right now, it's not working, things feel like they are moving very fast and this causes anxiety. as u've observed, my impression is that there's lots of sweeping claims made in papers that lack rigor or a deeper understanding. at first it's intimidating reading all of the works, like maybe u might pull out 30 that have been published in the last year for niche stuff in 3d gan inversion. u think ur absolutely blasted and finished. but read them all a few times, anxiety goes away, and you notice they all have the same issues. furthermore, u run their code, and you notice more issues with the models than u saw in the paper. it's recently been my experience in computer vision. so, things are often dressed up in ML reseach, it's not scholarly and i loathe this. but u get good at intuiting likely areas of weakness by noticing common patterns. part of doing phd is learning to sit with that anxiety. above all else you also have to always always always double triple check no one else has done your idea, don't be seduced by your curiosity without checking if it would be publishable...hwoever it sounds like for u this new publication happened randomly. if you can fix the problems with their method, then you have a paper, this could be a very good opportunity, particulary if you could easily show it fails for higher dim case on a synthetic dataset. if u can't get a satisfying proof, u need to change approach to something tangentially related. sounds prosaic but we often struggle with this cos the feeling is very frustrating. however pivoting is the most important skill u will develop. we fall in love with our ideas and our theories but these work out to nothing without strategy and ability to let go / change focus. lastly, where did the paper come from? with these things u need to find out the context - who are the researchers, where are they working, how long have they been working together and so on. if u read previous publications from the authors (often v easy cos its the same ideas just less developed) u can get a feel for persistent problems that they haven't fixed. i think right now everyone is having a hard time getting publications in, like ppl in academia working in ML, idk if it will get better for the average punter but utilise your social knoweldge to the fullest extent you can - ask ur supervisors how they find the current situation and what their thoughts are. if u can cop this idea not working out, that's great - often most ideas don't work. but then one does. and then u have a phd. and then u fix the idea that broke 2 years ago. these things never happen instantly, despite impression of increased publication frequency. good luck!