NoIdeaAbaout

There are much less alternatives, you could use keras in R but is actually a hassle. There are few alternatives for topic modelling in R:

• the classical Latent Dirichlet allocation (LDA), you can use with a specific R library called: topicmodels, here you can find more information: https://www.tidytextmining.com/topicmodeling
• You have also a package to better visualize topics, https://github.com/cpsievert/LDAvis
• you can create something similar with BERT in R, I suggest to do something more simple: https://blogs.rstudio.com/ai/posts/2019-09-30-bert-r/
• if you really want you can use python scripts in a r script

If I would start again I would focus more on NLP, many of the things I have studied are obsolete. my only suggestion is: to choose the field you like more and go in-depth, especially the application field. For example, I work on AI in biological applications, beyond the knowledge of the algorithms, domain expertise is key

That's very small for a trasformer, as a rule of thumb, this is meaning 25M parameters. Not sure there are similar ones

you can try this:

https://arxiv.org/pdf/2006.03236.pdf

you can also use this, is very simple library:

https://maartengr.github.io/BERTopic/index.html

I think you can try a similar way for another task, for me, the approach can be generalized to different tasks

Have you seen this article by Google?

https://arxiv.org/abs/2305.02301

https://blog.research.google/2023/09/distilling-step-by-step-outperforming.html

they claim that they were able to distill for reasoning task PaLM with T5 (2000 times difference in size) and the distilled T5 was outperforming PaLM

code is here:

https://github.com/google-research/distilling-step-by-step