16

(1) What all features should I extract? First, realize that you're not classifying documents. You're classifying (document, query) pairs, so you should extract features that express how well they match. The standard approach in learning to rank is to run the query against various search engine setups (e.g. tf-idf, BM-25, etc.) and then train a model on the ...


7

At its core, a search engine index is simply an index that supports full text search. The most simple way to do that is a simple inverted index, i.e. for each word that occurs in any of the documents you have indexed, store a list of references to all the documents that contain this word. For a university project, that's probably enough, but of course ...


5

So, when a GET request is sent over HTTPS, its query parameters are secure in transit. However, it can more easily leak data at the endpoints (browser history, referrer URLs, and server logs notably) than POST can. See this answer and this old blog post. Aside from referrer leakage, I'm usually not too concerned about using GETs. GET URLs are nice precisely ...


4

I'm in favor of a tagging system over predefined fields when many of them are just going to be N/A. All predefined fields do in those cases is suggest things to think about adding. I don't need a pile of text boxes to suggest a list of things to consider adding. I can use lists, tag clouds, or just a paragraph for that. However, a user defined tagging ...


3

Questions like this usually boil down to a trade-off between memory usage and speed. A program which uses a highly packed representation for keeping boards in memory will always require some time to unpack the data into a form where it can evaluate and apply the game's semantics. This is especially true when this "unpacking" is done implicitly, by ...


3

Path- finding algorithms such as A* routinely do this using some cost function (e.g. Distance, time, or even waiting-time) and heuristics to select the most promising candidate for expanding the path (e.g. If possible departure from same station/airport, and distance between arival point and target). The basic schema would be: use a sorted queue pick the ...


3

Hash the words to a key value pair of (word, set of documents). When you do the search, insert the sets found into the hashtable. Then do a union on the sets for OR and intersections for AND


2

I would consider using a standard keyword search with the nouns and verbs from your query as a way of generating a shortlist of possible results and then using an NLP parser (e.g. Stanford Core NLP) to preform a more detailed analysis on each contender in order to filter them to only exact matches. Assuming a reasonable corpus size and that the queries use ...


2

Short answer: you need both fake data, with well defined input X and output Y real-world data, probably with the modifications you suggested Use the first one especially when doing TDD (as your tag indicated), and after you have the basic algorithm ready, use the second kind of data for integration or acceptance tests. The first kind of tests will prevent ...


2

A search engine would normally use an inverted index in order to be able to efficiently search through large amounts of data (the same principle is used if the documents come from a source different than a web crawler). You split the text into words, and for each word the inverted index contains IDs of the documents which contain that word. In order to ...


2

I would somewhat disagree with @Ewan. While yes, you can wind up with a big blob of unstructured data, that’s only an inevitable consequence if you don’t work to prevent it. In counterpoint, I’d argue that the public internet is actualy a collection of microservices, most of which happen to serve HTML documents of one flavor or another. Google, Bing, etc ...


2

I Would avoid a 'Central Service' you can end up with a big blob of unstructured data. Presumably once the user has found something in there, you will want to do something with the result and this will require structured data. Each MicroService should be responsible for its own data and supply a search function. When the user does a global search, you ...


2

I'd suggest you use ElasticSearch highlight feature or taking a look at how it works internally. With settings, it supports breaking a text into fragments and find the best fragments among all fragments. Finally it highlight only those terms that participated in generating the hit on the document.


2

Best way? No clue. It depends on what you are using this encoding for. For example the best way would for a human would be column/row to column/row. For a computer? Considering that you have a 64bit board it takes exactly 6 bits to identify a location. Even then extra information such the piece being white/black bishop reduces that to 5 bits. 12bit moves, ...


2

You want to store enough information to make the move and to undo the move. So that is start position, end position, piece taken, and piece converted to at the very least. 6 bit + 6 bit + 3 bit + 2 bit. If you want to take it to 16 bit at most, store 1 bit for "conversion" set if a pawn moves to the last row, and you can leave out either the 2 bit ...


2

This is a very hard problem. You are asking for an Artificial intelligence piece of software to fully comprehend the target language's semantics and culture in order to determine the results' relevance. There are a few workarounds: Use Google/Bing/whatever other search engines exist out there for learning and testing. In effect, you would try to duplicate ...


1

We are building a new product in real estate space and the end users of this product are not so tech savvy. To have better user experience with our product, we want our users to find relevant things quickly and easily. Apart from a simple UI, a universal search bar seems to add value. The search bar with auto-complete will allow users to find information ...


1

Comparison with an oracle is always a good idea, when available. Comparison with a principle competitor is also a good idea. Your metric for comparison doesn't appear well thought out (what if the results come in a much different order? What if they come in a slightly different order?). If you are indexing one set of URLs (documents) and your oracle (...


1

Depends on your distribution of indices: Thake the following example sentence: The government passed a bill on tax deductions If every single word would be assigned an index and a context you would be storing the context quite a number of times (assuming there are 4 keywords in this sentence (the nouns), you would probably end up storing the whole ...


1

You can store the index and the first couple of highlights and its surrounding text per item(as their preview). By this way looking at the initial results user can decide to retrieve the document or not. This approach will definitely increase your storage but not as much as the your first approach. It will be though more or less at the same time if your ...


1

You could go with . Dictionary<string, HashSet<Int>> se word docID But I thought you had to build from scratch I don't know java It may be called a HashTable in java var docs = se["my"].Interset(se["university"]); var docs = se["my"].Union(se["Delhi"]);


1

Stem, but sort by relationship. So exact matches come first, then close lexical pairs (programmed vs. programming, programmer), then stems, then entirely different forms (programmatic, programmatical...). English suffixes are fairly consistent, so while you'll need to store your data across two related tables (for the sake of the answer I'll assume your ...


1

It can be one of the following: Storing to the database, and mapping URL paths to database queries Storing to a file, and mapping URL paths to file names Storing to memory, and mapping URL paths to data structures In terms of the URL design, it is stateless: Imagine a shop where rather than being self-service, there is a shopkeeper who upon being asked, ...


1

You implement a form of microservice architecture - each of the 25 sites has a 'front end' (to them) that is a back end to your website. You make requests to each of these which make the appropriate request to the sites. Its up to you how to aggregate the results you get back, whether to take them as they come, or to hold them and process them into a single ...


1

Rather than centralising the date overnight, why not use a user interface, which naturally shows the progress on each of the sites being searched. https://iwantmyname.com/?domain=somedomain is a good example interface for domain name search. You would basically set up a user interface with 25 rows, which contain the name of each of your databases. The ...


1

SQLite has a full text search extension. I suggest you take a look at the documentation. It could optimize the searches because it would create an inverted index that maps from each unique term or word that appears in the dataset to the locations in which it appears within the table contents. Such searches should be quicker than just issuing SQL comparisons ...


1

I'm not entirely sure I've understood correctly, but I think you can reduce your 7 database accesses to 1. Encode each security level as a bit in an integer. Store the security level with each record. Establish which security levels a user is allowed to see - encode this as a similar integer. When searching for records, include a requirement that the ...


1

If I understand correctly, previously the search results from the engine was fast. Now you do that, but additionally filter the results afterwards with up to 7 DB queries and intersection filters. Obviously, the problem lies with the DB queries, or the intersection sorting and filtering! The next question is - which is slow? Is the DB a slow box, are you ...


1

There are also some tweaks you can apply on server configuration, in order to reduce split-brain problems in large(ish) clusters, namely the so-called discovery.zen.* parameters. There is not that much information about the inner workings and algorithms of ES, but what I found during my evaluation was the reference documentation about the server setup. In ...


1

There is not an incredible amount of detail in the online Elasticsearch: The Definitive Guide, but suffice to say, with respect to: Cluster membership: the oldest node becomes the "master" and it appears some sort of gossip protocol is used when nodes join and leave the cluster. A leadership election does occur when the master leaves the cluster (which I ...


Only top voted, non community-wiki answers of a minimum length are eligible