We are writing a search engine from scratch in a quite uncommon language, Aramaic, mostly for learning purposes but also because few resources are available in given language.

The engine is/will be written in Python, and:

  1. It is a human language search engine.
  2. It's not about paying someone to do the tests, but trying to find a way to resolve the issue programmatically (if even possible) for learning purposes mostly.

We would like to test automatically if the search engine works efficiently but without knowing the language we were wondering if there were maybe tools or ways to test it get statistics. If we had users who spoke the language we could ask them to evaluate the search results but not having that, what are the other ways to test a search engine?

  • 1
    Can you clarify whether you mean a human language or a programming language?
    – Andy Hunt
    Commented Jul 11, 2014 at 7:56
  • 2
    The most effective (and probably affordable) way is to hire some native speakers as testers.
    – Doc Brown
    Commented Jul 11, 2014 at 8:10
  • If there was an automated tool; wouldn't that mean that such tool would get its data from another search engine in that a language?
    – SJuan76
    Commented Jul 11, 2014 at 8:16
  • I have updated the question after your comments (EDIT2), thanks
    – vallllll
    Commented Jul 11, 2014 at 9:37
  • 1
    Make your software free software, and make it available very early. Publish its existence, ask researchers working on Aramaic (and people knowing Aramaic, at least as a liturgical language) to use it. Commented Jul 11, 2014 at 12:13

1 Answer 1


This is a very hard problem. You are asking for an Artificial intelligence piece of software to fully comprehend the target language's semantics and culture in order to determine the results' relevance. There are a few workarounds:

  1. Use Google/Bing/whatever other search engines exist out there for learning and testing. In effect, you would try to duplicate the ranking algorithm in a 'black box' fashion, using statistical methods to see whether the results 'align' according to some error metric.

  2. Advertise for target language speakers online (Mechanical Turk or any other outsourcing marketplace), always checking whether the reviewers really speak kiswahili or whatever language you need.

EDIT: since this is for texts in Aramaic (I presume Old Aramaic), I'd suggest looking at Biblical Hermeneutics.SE (hat tip to TRiG), Christianity.SE and Mi Yodeya.SE for links to online communities. There are quite a few people who are very well versed in the language.

  • 1
    Also Biblical Hermeneutics SE.
    – TRiG
    Commented Jul 11, 2014 at 11:40
  • @TRiG : Thanks, left this one out. Should have been the first hit. Commented Jul 11, 2014 at 11:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.