Zahlii

Even in 3D space, the number of vectors in a given angle to another vector are infinite, so which would you pick in an n dimensional space? In general calculating the angle between two vectors is a loss-introducing function (you take two sets of n numbers and condense it down to one). You can narrow it down to a set of N-1 linearly independent vectors that form a base of the given set of vectors in a given angle. Somebody with more linear algebra knowledge feel free to correct me.

A plot with the fitted ln curve overlay would be helpful to check goodness of fit. It looks like 30-45 is actually somewhat linear and not ln, and without domain knowledge about what x and y are I find it difficult to propose curve fits about what looks to be a piecewise function

If it’s just about classifying documents nothing stops you from iteratively hand labeling a new set of documents, training a model to suggest classes, correcting the model predictions on a new set of documents and repeat. Will be cheaper than using complex models and may get you there. If it’s about making the whole stuff indexable best bet is to use a vector database, sentence transformer embeddings and then putting the top 4-5 closest paragraphs to your search into a LLM prompt for reranking. If it’s about extracting structure information from each document you will most likely need to use a) specialized custom model or b) one LLM prompt for each document

Who set up the score of 0.97 as a goal? Are you sure it is attainable given the data? Have others provided kernels/notebooks that attain these scores? In most cases it’s not the lack of modelling on your side but rather lack of data on the other.