Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [natural-language-processing]

Natural language processing draws knowledge from a diverse collection of fields including computer science, linguistics, and statistics in order to extract pertinent information from the spoken or written word.

2
votes
0answers
80 views

Software design strategy for a machine learning tool that outputs a subset of the text input (Information Extraction)?

Let's say I have thousands of pdfs that are each about 30k words written in conversational English. In each of the pdfs there is a name / names of a person/people who snowboard. There are also many ...
3
votes
1answer
81 views

What approaches can I take to figure out the “relevancy” of certain terms in a string?

I'm not even sure "relevancy" is the most accurate word, so I'll just describe the problem: I'm building an app that needs to somehow parse product descriptions from a popular website (let's just say ...
0
votes
2answers
76 views

Divergent Bodies of Truth, Relationships of Concepts

A recent endeavor of mine requires asking users how related two concepts are. Example: Kiwi and Fruit. Identical? Fairly Similar? Rather different? Unrelated? It occurs to me that for certain ...
0
votes
1answer
212 views

What is the best way to summarize a sentiment value for a paragraph of text based on the sentiment value for the sentences within it?

So I am using Stanford CoreNLP in my project. I have data which consists of reviews of products on a forum. I need to be able to assign a sentiment value to a given review. CoreNLP allows you to ...
1
vote
2answers
278 views

Automatic summarization - whole text or paragraphs

I am writing a tool that will give users the ability to summarize text content on a webpage, by highlighting the text that they wish to get summarized. So far, I've received results that I can work ...
3
votes
2answers
450 views

How to build a class for comparing words in a lexical dictionary?

I'm wanting to create a class that stores words in a set so that I can see if a word belongs to that set or not. I'm not wanting to build ever set every time I instantiate the class, so I'm using what ...
3
votes
2answers
207 views

Is polymorphism appropriate for modeling natural language form (structure), or is there something better?

Let's take French and Japanese adjectives as simple examples. French adjectives have gender (masculine or feminine) and number (singular or plural), whereas Japanese nouns have neither. However, ...
0
votes
1answer
102 views

Which strategy to use in Client-Server messaging to trigger specific behavior in them?

I'm developing a virtual assistant to manage transactions in a business. I'm using an external service to handle Natural Language and get intents, actions and parameters from my user requests. The ...
2
votes
2answers
980 views

How to create an efficient implementation of a dictionary of synonyms?

How do I create a dictionary of synonyms that is efficient in terms of retrieving synonyms of a word? Basically, I want to use some sorts of data storage to store groups of words, and enable an end-...
1
vote
1answer
419 views

What type of “spam filter” algorithm will tokenize characters for non-exact matches?

I was having a look at a basic example of spam filtering using a logistic regression algorithm and an answer posted on this Stack Overflow question: https://stackoverflow.com/questions/29291263/mllib-...
1
vote
1answer
55 views

Extract the main entity from a body of text for text categorisation?

I'm trying to categorise products based on various fields of data. I've had some success just matching search terms in the product names, but this naive approach doesn't work when it comes to larger ...
16
votes
2answers
604 views

How to find hard to misspell given names?

Here is a question that I believe could be solved with some data mining and a sophisticated algorithm, but I don't quite know how. Any pointers as to what data sources to use and what algorithm to ...
6
votes
1answer
230 views

Identifying plagiarized jokes? [closed]

I'd like to be able to identify duplicate jokes posted on a website. I can build up a reasonably large database of previously-posted jokes, and then I'd like to look at each new joke as it comes in ...
-3
votes
1answer
259 views

Using NLP can we create a summary of functional specification [closed]

Just thinking that is it somehow possible by using Language processing APIs to extract a must have points or an intelligent summary from documents like functional specifications ..
1
vote
4answers
1k views

How to automatically connect letters into words? [closed]

I would like to connect letters into meaningful words? E.g. input: T E ST IN G WE B A P P LIC AT IO N S output: TESTING WEB APPLICATIONS what kind of task is this? A task in NLP, or just some ...
8
votes
2answers
956 views

How do personal assistants typically generate sentences?

This is sort of a follow up to this question about NLG research directions in the linguistics field. How do personal assistant tools such as Siri, Google Now, or Cortana perform Natural Language ...
0
votes
1answer
126 views

What is latent parsing in NLP?

I'm reading a paper describing a NLP work, and I hardly catch the concept of a term, "latent parsing". Original paper: http://web.stanford.edu/~angeli/papers/2013-acl-temporal.pdf The figure in the ...
4
votes
1answer
141 views

How can I test a search engine for an uncommon human language?

We are writing a search engine from scratch in a quite uncommon language, Aramaic, mostly for learning purposes but also because few resources are available in given language. The engine is/will be ...
0
votes
2answers
208 views

How could an expert system get some meaning from a sentence? [closed]

How could an expert system tell following scenarios apart? How could we include some meaning of a sentence, if at all? A machine needs 1 day to produce x units, and, we know that two machines like ...
0
votes
1answer
385 views

Natural Language to Search Criteria - Date Ranges

Consider an application that stores a set of records that contain: Description Cost Purchase Date I'd like to be able to allow users to utilize natural language to search the dataset. For example, ...
5
votes
6answers
4k views

Are there any algorithms for splitting or combining words into their more common form?

Are there any existing algorithms which can look through a list of words and split or combine words into their more common form? For example, I have a list of many business names in the health care ...
0
votes
1answer
1k views

What is the purpose of chunking strings in natural language processing?

I see many examples of libraries that use a 'chunker' and also people asking how to write chunkers, but what is this for and why do we need it? Isn't it enough to split a text by whitespace or other ...
4
votes
1answer
436 views

Sentence Tree vs. Words List

I was recently tasked with building a Name Entity Recognizer as part of a project. The objective was to parse a given sentence and come up with all the possible combinations of the entities. One ...
1
vote
1answer
210 views

How to do simple natural language categorization [closed]

How would you go about parsing a sentence like "Bought two kilos of steak from Acme supermarket" into a data structure like the following JSON representation: { item: {name: "steak", tags: "meat,...
2
votes
1answer
338 views

How can I distinguish between one-worded expressions and two-worded expressions?

I was told to create two lists concerning most frequently used words from a plain text (10 Mb arbitrary texts) as monograms (for single worded expressions such as human, water, is) and bigrams (for ...
3
votes
1answer
236 views

Tasks incorrectly categorized with online text classifiers

Context: Finding company to do the job The use case of an application I want to write is that a user will describe a task to be performed by some company, e.g., I want to move 10 boxes (30x40x50cm)...
3
votes
3answers
1k views

NLP - Queries using semantic wildcards in full text searching, maybe with Lucene?

Let's say I have a big corpus (for example in english or an arbitrary language), and I want to perform some semantic search on it. For example I have the query: "Be careful: [art] armada of [sg] is ...
12
votes
2answers
2k views

Persisting natural language processing parsed data

I've recently started experimenting with natural language processing (NLP) using Stanford's CoreNLP, and I'm wondering what are some of the standard ways to store NLP parsed data for something like a ...
3
votes
2answers
4k views

Separating words in a string

How do I separate words in a string? In the following I have a random sample of words in a string extracted from text file with over a million words. Here's the string: "intervene Pockets ...
1
vote
2answers
2k views

How to create a Semantic Network like wordnet based on Wikipedia?

I am an undergraduate student and I have to create a Semantic Network based on Wikipedia. This Semantic Network would be similar to Wordnet(except for it is based on Wikipedia and is concerned with "...
10
votes
3answers
6k views

What algorithm(s) can be used to achieve reasonably good next word prediction?

What is a good way of implementing "next-word prediction"? For example, the user types "I am" and the system suggests "a" and "not" (or possibly others) as the next word. I am aware of a method that ...
3
votes
2answers
590 views

Available options for classifying words in text?

I am researching ways to classify words in text and I'm wondering what options there are and which are best suited to this job. I'm mostly interested in keywords which are most often nouns. So far I ...
3
votes
4answers
4k views

Programming a chatterbot - understanding language [closed]

I hope this is an answerable question. Let me give it some context: I am a psychology student and a programmer. I'm going to look into creating a artificial intelligence in the form of a chatterbot. ...
2
votes
1answer
3k views

Is Agile Development used in Machine Learning and Natural Language Processing?

I've been developing web apps for a while now and it is standard practice in our team to use agile development techniques and principles to implement the software. Recently, I've also become involved ...
7
votes
2answers
174 views

Guess if a time is AM or PM

I'm currently in the process of writing a human date parser. By human date, I mean it should be able to interpret strings as "tomorrow at 2" and return a valid date depending on the current time. The ...
11
votes
6answers
3k views

How to teach a script to detect sarcasm? [closed]

I'm currently building a fun script, that basically matches given phrases and gives a predefined response based on the match-points. You can ask it to retrieve some information based on live feeds, ...
142
votes
14answers
9k views

Simple method for reliably detecting code in text?

GMail has this feature where it will warn you if you try to send an email that it thinks might have an attachment. Because GMail detected the string see the attached in the email, but no actual ...