How can I test a search engine for an uncommon human language?

Question

We are writing a search engine from scratch in a quite uncommon language, Aramaic, mostly for learning purposes but also because few resources are available in given language.

The engine is/will be written in Python, and:

1. It is a human language search engine.
2. It's not about paying someone to do the tests, but trying to find a way to resolve the issue programmatically (if even possible) for learning purposes mostly.

We would like to test automatically if the search engine works efficiently but without knowing the language we were wondering if there were maybe tools or ways to test it get statistics. If we had users who spoke the language we could ask them to evaluate the search results but not having that, what are the other ways to test a search engine?

Answer 1

This is a very hard problem. You are asking for an Artificial intelligence piece of software to fully comprehend the target language's semantics and culture in order to determine the results' relevance. There are a few workarounds:

1. Use Google/Bing/whatever other search engines exist out there for learning and testing. In effect, you would try to duplicate the ranking algorithm in a 'black box' fashion, using statistical methods to see whether the results 'align' according to some error metric.

2. Advertise for target language speakers online (Mechanical Turk or any other outsourcing marketplace), always checking whether the reviewers really speak kiswahili or whatever language you need.

EDIT: since this is for texts in Aramaic (I presume Old Aramaic), I'd suggest looking at Biblical Hermeneutics.SE (hat tip to TRiG), Christianity.SE and Mi Yodeya.SE for links to online communities. There are quite a few people who are very well versed in the language.