Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 11 months ago

MODERATORS

communick@academy.garden

[D] Exact kNN for high dimensional data (alien.top)

submitted 10 months ago by xero786@alien.top to c/machinelearning@academy.garden

7 comments fedilink hide all child comments

Hi guys, I am new to ML and was experimenting with kNN search algorithms.

I have very high dimensional data 1000+ dimensions. What data structure is best suited for such high dimensional data.

I can bring down the dimensions to apprix 150 using using PCA without being too lossy.

Even then I am having hard time finding techniques that work with such high dimensional data. I am not looking for Approximate NN search using LSH.

What is the best technique that can be used here, kd tree doesn't work well with high dimensional data, would Rtree or ball tree be a better choice or something different?

you are viewing a single comment's thread
view the rest of the comments

[–] xero786@alien.top 1 points 10 months ago (1 children)

My dataset is about 8000 points, and the reason I am not using ANN is that I am trying to study and experiment how exact kNNs work, what can I do with them, what's best amongst them in high dimensional space...

[–] resented_ape@alien.top 1 points 10 months ago

I understand your investigative spirit but I think you are going to discover that all methods of finding exact kNNs are different flavors of "slow" at high dimensions. There might be some exotic variations that can shave off some of the constant factor but they are usually restricted to Euclidean or inner-product-like distances (although that's usually what people want). FWIW 8000 points doesn't seem like a huge dataset.

Here's some R code and timings using the FNN package which has a few options:

data8k &lt;- matrix(rnorm(n=8000 * 1000), nrow = 8000)
system.time(knn &lt;- FNN::get.knn(data8k, k = 30, algorithm = "brute"))
   user  system elapsed 
  70.19    0.06   75.87 
system.time(knn &lt;- FNN::get.knn(data8k, k = 30, algorithm = "kd_tree"))
   user  system elapsed 
  78.51    0.14   85.08 
system.time(knn &lt;- FNN::get.knn(data8k, k = 30, algorithm = "cover_tree"))
   user  system elapsed 
 129.52    0.14  134.01 
system.time(knn &lt;- FNN::get.knn(data8k, k = 30, algorithm = "CR"))
   user  system elapsed 
  70.41    0.08   74.40

That's single-threaded on my very-much no-longer-impressive laptop. The fact that brute force search does so well in comparison to the others suggests that there aren't good options for your data.