It's the same mathematically but not computation wise, the tokens are projected to a smaller dimension. The complexity is 2Nd whereas it'd be N² if you'd fuse the weight matrices.
It's the same mathematically but not computation wise, the tokens are projected to a smaller dimension. The complexity is 2Nd whereas it'd be N² if you'd fuse the weight matrices.