Hot answers tagged

15

(1) What all features should I extract? First, realize that you're not classifying documents. You're classifying (document, query) pairs, so you should extract features that express how well they match. The standard approach in learning to rank is to run the query against various search engine setups (e.g. tf-idf, BM-25, etc.) and then train a model on the ...


7

Python has a great natural language toolkit, the NLTK. It supports word tokenisation out of the box: >>> import nltk >>> input = 'We have some great burritos!' >>> tokens = nltk.word_tokenize(input) >>> tokens ['We', 'have', 'some', 'great', 'burritos', '!'] >>> nltk.pos_tag(tokens) [('We', 'PRP'), ('have', 'VBP')...


4

I have only experience with Solr and Sphinx, so can't really compare too much. And we don't use much 'fuzzy' search. But I worked with Solr a lot and think I know the docs quite well. First of all, the term 'document' is to be understood in a very technical way. By no means does this limit your search to typical text documents. We use Solr to search ...


3

Yes, Solr supports out-of-the box (well, after a bit of configuration, see the examples from version 4.9 onwards) PDF and Word documents. The thing to note is that Solr != Lucene. Solr is a higher level abstraction over Lucene, and as such it has a different API, features and behaviour. IMHO, the difference between Solr and Lucene utilisation can briefly ...


3

The main downside I see is updates. The only way to do an update in the Lucene ecosystem was to read the whole document, modify it, delete the original, and write back the contents as a new document. Some syntactic sugar has been added to Solr to make this easier. Lucene itself doesn't support per-field updates, though some stacked update feature appears to ...


3

Yes. I've done projects in the past where we essentially used lucene as a data store in lieu of a database. This was long before NoSQL was hot. Really there's no fixed definition of what qualifies as a NoSQL database, so anything that stores and retrieves data is sufficient. Things like dbm files have been around forever.


3

I would recommend starting with Solr, then do your machine learning with Mahout and Hadoop. Solr will give you basic text analysis through word stemming, normalization (lower-casing), and tokenization. If you enable term vectors in the schema you can feed those directly into Mahout and experiment with the different algorithms there. A lot (maybe most) of ...


3

Well, the best thing you can do is run your own benchmarks! Compare the average speed over several thousand test queries on an index that includes ngrams vs one that excludes them. It doesn't have to be your full actual index (since that may take a long time to generate), just a large enough sample size to get an idea. Note that you can use debugQuery=on ...


2

I would consider using a standard keyword search with the nouns and verbs from your query as a way of generating a shortlist of possible results and then using an NLP parser (e.g. Stanford Core NLP) to preform a more detailed analysis on each contender in order to filter them to only exact matches. Assuming a reasonable corpus size and that the queries use ...


2

Lucene indexes don't have to be recreated from scratch each time they're changed. Much like your database tables, you can insert new records, delete old ones and affect an update using a delete/add pair. Most databases have a mechanism which will allow calling external code, and you can leverage this in triggers that run on INSERT, UPDATE and DELETE to ...


2

TL;DR: You can do things faster. I used to have a similar problem. Turns out, with Lucene, rebuilding the entire index using code optimized for it, starting from the scratch, using a SELECT field1, field2 FROM table query with no where clause, was quite fast. The reason for the quick execution time probably was that the Lucene-interfacing code didn't need ...


1

Comparison with an oracle is always a good idea, when available. Comparison with a principle competitor is also a good idea. Your metric for comparison doesn't appear well thought out (what if the results come in a much different order? What if they come in a slightly different order?). If you are indexing one set of URLs (documents) and your oracle (...


1

Even if you introduce something like your version field into the documents, you cannot know in advance what kind of bugs will be there in the future. There can always be one which makes it necessary to rebuild the whole index. So you should look for a strategy which makes this possible. You wrote Recalculating the complete index requires multiple hours ...


1

PostgreSQL supports both JSON data type and GIN indexes out of the box so that will fly just fine with relational databases. You can also choose to model some of the fixed fields as normal columns and only use JSON for the dynamic fields. I would favor PostgreSQL over MongoDB especially if you also need proper ACID transactions and if your data fits on one ...


1

It sounds like the biggest problem you have left is to actually develop the categories and flesh them out. Given a set of categories, and a set of 'marker' words for each category. (You'll want to read about stemming (turn vegetables -> vegetable) and stopping (skip common, meaningless words the, etc). A good implementation of the stemmer you can find will ...


1

As you use the terms NP and VP, I think you are talking about syntactic, not semantic. There are differences between the two. You can check out dependency grammar to see how it is different from CFG syntactic grammar. I think if you use wildcards for semantic search, people would consider that a 'hack'. This will be what I would do: To properly do ...


Only top voted, non community-wiki answers of a minimum length are eligible