We’re rewarding the question askers & reputations are being recalculated! Read more.
15

(1) What all features should I extract? First, realize that you're not classifying documents. You're classifying (document, query) pairs, so you should extract features that express how well they match. The standard approach in learning to rank is to run the query against various search engine setups (e.g. tf-idf, BM-25, etc.) and then train a model on the ...


13

There is a Google Custom Search page set up for Go Information "for everyone complaining that it is impossible to search for Go docs and info." I've also gotten good results through Google by including "go lang" as the first part of my search term. I do notice that I don't get a lot of independent blogs in my search results, but I do get results from Stack ...


7

At its core, a search engine index is simply an index that supports full text search. The most simple way to do that is a simple inverted index, i.e. for each word that occurs in any of the documents you have indexed, store a list of references to all the documents that contain this word. For a university project, that's probably enough, but of course ...


7

Python has a great natural language toolkit, the NLTK. It supports word tokenisation out of the box: >>> import nltk >>> input = 'We have some great burritos!' >>> tokens = nltk.word_tokenize(input) >>> tokens ['We', 'have', 'some', 'great', 'burritos', '!'] >>> nltk.pos_tag(tokens) [('We', 'PRP'), ('have', 'VBP')...


7

One can also use "golang" as a search term. It works fine to catch the content from the Go mailing lists.


6

Google (and I suspect Bing as well) have gotten much better at reading and indexing text found in JavaScript elements during the past 3-5 years or so. They do this for two reasons. First, to provide better indexing of content for users and, second, to detect and thwart various spamming techniques. The problem is that you may not get indexed as well as you ...


4

Here are some resources for you to peruse. In short: If you have dynamic generation schemes embedded in your javascript (javascript menus for example), provide a static sitemap to the embedded content. Google can crawl Ajax links, if you take certain steps. https://developers.google.com/webmasters/ajax-crawling/docs/getting-started http://www....


4

Ok, how do we start. First of all this is written based on Lucene 3.6. NRTManager is used for Near-Realtime-Scenarios where Writing and Reading is very close after each other. An Example would be Twitter (that is actually using a modified Version of Lucene). In these cases you are not suppost to close your IndexWriter as all changes that occure are being ...


4

Actually you're more looking to find a "result ordering" rather than a search algorithm. Anyway, I would grade the results using a points mechanism. This makes it easier to handle, easier to maintain and to enhance if you wish to. Moreover, it also helps you handling the case of multiple keywords search. This is how it works: For each keyword the user ...


4

So, when a GET request is sent over HTTPS, its query parameters are secure in transit. However, it can more easily leak data at the endpoints (browser history, referrer URLs, and server logs notably) than POST can. See this answer and this old blog post. Aside from referrer leakage, I'm usually not too concerned about using GETs. GET URLs are nice precisely ...


4

I'm in favor of a tagging system over predefined fields when many of them are just going to be N/A. All predefined fields do in those cases is suggest things to think about adding. I don't need a pile of text boxes to suggest a list of things to consider adding. I can use lists, tag clouds, or just a paragraph for that. However, a user defined tagging ...


3

As others have commented on your question, you need server-side functionality to provide effective and efficient search in a website; otherwise you would need to make all searchable content available to the client-side search engine; unless the volume of searchable content was very small, this would be troublesome. Once you have the server-side ...


3

As a general rule, search engines do not crawl content generated by JavaScript or Ajax. The most notable exception being Google's crawlable Ajax. But that only applies to Google and even then that's a bad idea. To make your site more search engine friendly you will need to make content available via static links. Without these you essentially have a one ...


3

I'm surprised there's hasn't been an avalanche of replies of web developers, to the effect of: "When we develop web pages, half the effects don't work in various versions of IE, especially the older ones without adding various tricks and hacks. The other browsers - Firefox, Opera, Chrome, etc have been much better about adhering to and supporting the ...


3

As others have pointed out, golang (and its variants) seems to work at the moment. I'll also add that I don't think we should have to pick names of products, projects, or services based solely on search optimization. Other one, two, or three letter languages seem to be rather searchable. "C" isn't exactly the best name for a language but it seems to be ...


3

The freely distributed draft of Introduction to Information Retrieval is going to be your prime reference material. It handles search (information retrieval) from basic to advanced level.


3

Path- finding algorithms such as A* routinely do this using some cost function (e.g. Distance, time, or even waiting-time) and heuristics to select the most promising candidate for expanding the path (e.g. If possible departure from same station/airport, and distance between arival point and target). The basic schema would be: use a sorted queue pick the ...


2

It's rarely impossible or even all that difficult to implement a progressive enhancement approach that delivers static content when JavaScript is turned off. This isn't just an accessibility win. It's also mighty handy as a fallback for mobile browsers whose JS interpreters simply aren't worth attempting to normalize for (*cough*blackberry*/cough*). I'd ...


2

the safest bet is to have the content you want spidered as HTML (not brought through ajax). Even if Google does something, that doesn't mean other spiders do. If uncertain, you can use Google webmaster tools to see how Google sees your page ( https://www.google.com/webmasters/tools/home ) under diagnostics/fetch as googlebot


2

Building a Search Engine This is an introductory course to CS that's going to start on the 20th, I suggest you check it out, it's offered free of charge.


2

Generally, a website like an e-commerce website, and usually also news websites, blogs and anything else with highly structured pages (such as Stack Overflow itself -- though don't try looking for it), will publish an XML sitemap file. An XML sitemap is basically just a big XML file with a bunch of links to all of the pages in the site. It usually also ...


2

We had EXACTLY the same needs that you are describing. We are a python shop... we sell around 300,000 unique items - so we needed fast updates. After testing Xapian I decided against it. What we ended up with was PyLucene - we have a server that is constantly rebuilding search indexes and replaces copies of read-only search indexes periodically. When you ...


2

Why not let the text search engine like lucene or sphinx do all the work? I use Lucene and have found it to be very good at searching multiple tables and fields for whatever you set as a target. You don't say what your web front end is written in but both of these tools can be used with most languages. You will be adding a lot of code to maintain in the ...


2

First, Google actively participates in the "science" of technology and often shares their knowledge by releasing papers from their R&D department. You can find those papers from the reference link below. I haven't searched for specific papers on search retrieval algorithms but there should be more than enough information available on the subject from a ...


2

This area of study is known as Information Retrieval. This Wikipedia article contains a good summary and lots of useful links.


2

I find that "go language" seems to give fairly accurate results. If there is a specific topic tacking that onto the end should work fine.


2

This is a very hard problem. You are asking for an Artificial intelligence piece of software to fully comprehend the target language's semantics and culture in order to determine the results' relevance. There are a few workarounds: Use Google/Bing/whatever other search engines exist out there for learning and testing. In effect, you would try to duplicate ...


2

I think this is leaning more towards a user experience sort of question, but I want to mention one thing before I go into that. If you're searching things that you have already loaded into the page, then using JS will probably be your best bet since you'll be searching records that the server side has already fetched and given to the page. Now if that isn'...


2

I would consider using a standard keyword search with the nouns and verbs from your query as a way of generating a shortlist of possible results and then using an NLP parser (e.g. Stanford Core NLP) to preform a more detailed analysis on each contender in order to filter them to only exact matches. Assuming a reasonable corpus size and that the queries use ...


2

Short answer: you need both fake data, with well defined input X and output Y real-world data, probably with the modifications you suggested Use the first one especially when doing TDD (as your tag indicated), and after you have the basic algorithm ready, use the second kind of data for integration or acceptance tests. The first kind of tests will prevent ...


Only top voted, non community-wiki answers of a minimum length are eligible