I have the need to match incoming fragments of semi-structured text to previously encountered fragments.
The most text fragments are sized ~200 to ~4000 characters and contain both human-readable text (a few sentences at most) and machine-generated text - string and numberic codes, IDs, URIs etc.
I've used K-Means clustering with various distance measures with some success but it's too slow for large datasets (or maybe it's my implementation?) - ~1000 items get clustered in about 30 sec but 10000 take over 10 min to produce ~150 clusters.
I tried LSH/Minhash but the probabilistic nature of the hashes sometimes misses important tags and misplaces some of the fragments as a result, plus the hash calculation doesn't improve speed much for such small texts - the cost of calculating 300 hash values is not 0 and then the array of 300 values is in the vicinity of the number of "words" the fragments get broken into anyway.
What is the fastest clustering algorithm that would be suitable for the task? Ideally something that I could implement from scratch, not a ready software/service/package.
Idea what the input looks like:
[Timestamp] A package of type Box with ID 123456 was not successfully checked in. [FKFGSIGURE] 12345 ~\logs\checkin\17-08-01.log Host:184.108.40.206 Pod:somepodname <...more stuff here...>
[DateTime] Invalid access attempt at Door 123. Badge XYZ was declined access. Suspending badge for 5 minutes. 23456 ~\logs\checkin\17-06-01.log Host:220.127.116.11 <...more junk...>
[Date] [Time] Host: 18.104.22.168 restart failed