daking999

joined 10 months ago
sorted by: new top controversial old
[D]In transformer models, why is there a query and key matrix instead of just the product? in c/machinelearning@academy.garden
[–] daking999@alien.top 1 points 9 months ago

I don't remember the ref but I browsed a theory paper at some point that did consider that representation (the product explicitly), possibly with something like nuclear norm regularization to keep the rank low.