r_s_s_i_u

As you said, it depends but my to go has been Sentence transformersSBert due to its effectiveness. But if you have access to sufficient compute or it's for offline use case (i.e get embeddings once and just keep refusing them), embeddings from LLMs works well on most use cases

More than validation loss, try to test your model against a benchmark (after scaling up your model). That will help you decide if your model is actually better (when compared to other models of the same scale)