Questions tagged [text-processing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
24 votes
4 answers
42k views

How can I extract words from a sentence and determine what part of speech each is? [closed]

I want to write something that takes a sentence and identifies each word it contains and defines what part of speech each word is. For example Hello World, I am a sentence would return this ...
  • 259
9 votes
4 answers
409 views

How should I implement a command processing application?

I want to make a simple, proof-of-concept application (REPL) that takes a number and then processes commands on that number. Example: I start with 1. Then I write "add 2", it gives me 3. Then I ...
9 votes
7 answers
942 views

Algorithm for determining transactions among weekly data series?

I'm trying to develop a small reporting tool (with sqlite backend). I can best describe this tool as a "transaction" ledger. What I'm trying to do is keep track of "transactions" from weekly data ...
  • 141
8 votes
2 answers
2k views

What is best practice to handle whitespaces when letting the user edit the configuration, the name=value pairs?

For instance, you let the user define the notorious path variable. How do you interpret apppath = C:\Program Files\App? This looks like a programming language adopted practice to ignore the white ...
  • 1
6 votes
2 answers
199 views

How does the Arabic typographic layout system work at a high level?

I have some Arabic content that is justified according to western conventions. I justified it because it is justified in ancient sources: However, the way Arabic text justification works is by ...
  • 2,412
6 votes
1 answer
1k views

Finding occurrences of a useful words and phrases in strings

I am building an app that analyzes posts by people by pulling their Tweets and Facebook posts. I need to process all the posts and find useful phrases. What I mean by useful is that, any word or ...
6 votes
1 answer
321 views

Tools for modelling data and workflows using structured text files [closed]

Consider a case when I want to try some idea of an application. But I want to avoid investing a lot of effort in coding UI/work flows/database schema etc before I see that it's going to be useful to ...
  • 1,109
5 votes
1 answer
761 views

How advanced are author-recognition methods?

From a written text by an author if a computer program analyses the text, how much can a computer program tell today about the author of some (long enough to be statistically significant) texts? Can ...
4 votes
2 answers
522 views

Database structure for word co-occurrence frequencies in a large corpus

I would like to store the frequencies with which words co-occur with each other over a variety of contexts in a large (> 1 billion tokens) text corpus. I need to store the word pair, the type of co-...
  • 51
4 votes
1 answer
497 views

Sentence Tree vs. Words List

I was recently tasked with building a Name Entity Recognizer as part of a project. The objective was to parse a given sentence and come up with all the possible combinations of the entities. One ...
4 votes
1 answer
2k views

Text comparison algorithm using java-diff-utils

One of the features in our project is to implement a comparison algorithm between two versions of text and provide a % change between the two versions. While I was researching, I came across google ...
  • 2,617
4 votes
1 answer
924 views

How to process an endless XML data stream

There is an endless data stream of XML messages (and "heartbeats"), that I receive via a telnet connection and through a site-to-site VPN IPsec tunnel. I'm still pondering. What is the best/most ...
  • 839
3 votes
6 answers
13k views

Which programming language for text editing? [closed]

I need a programming language for text editing and processing (replace, formatting, regular expressions, string comparison, word processing, text analysis, etc.). Which programming language is more ...
  • 3,083
3 votes
3 answers
521 views

Why is quantity in software still written as "1 result(s)"?

Lately, I've been noticing that a lot of software, be it a website, a client application, or a video game, often write a representation of quantity as follows: "1 result(s)". Now, I can understand why ...
  • 67
3 votes
1 answer
172 views

Integrating TeX into a Java desktop application

Looking to integrate TeX equations in a TeX-agnostic fashion, suitable for either ConTeXt or LaTeX, into a Java-based desktop Markdown editor. The possibilities are numerous, but I'm not sure what ...
3 votes
2 answers
5k views

Separating words in a string

How do I separate words in a string? In the following I have a random sample of words in a string extracted from text file with over a million words. Here's the string: "intervene Pockets ...
  • 131
3 votes
1 answer
1k views

What method for storing a text file in memory (c not c++) would allow me to open any format(UTF-8, Binary, etc) and a file of any size?

My first thought here is to use a dynamic array, but I am looking for something better. Currently I have the text files open into "chunks". Every word or group of spaces makes up a "chunk". Then I ...
  • 329
3 votes
1 answer
133 views

Efficiently editing text files [duplicate]

I have been working with some fairly large text files containing about two million lines of text. I don't know the length of the content or the lines in advance, just the number of lines. I have been ...
3 votes
1 answer
2k views

Best Practice - XML To Excel

I've to read a big XML file with a lot of information. Afterwards I extract the needed information (~20 Points(columns) / ~80 relevant Data (rows, some of them with subdatasets) and write them out in ...
  • 133
3 votes
4 answers
3k views

Windows compatibility with Unix/Linux newline "\n"

A follow-up to Difference between '\n' and '\r\n'. It's been few decades since the schism was introduced. Nowadays, when documents are being exchanged over the internet, typically ...
3 votes
3 answers
1k views

Domain-specific language for text search/processing?

I work for an organization that does a lot of work with government data. We have a couple of different projects where we've abstracted out common text search/manipulation operations into reusable ...
3 votes
2 answers
2k views

Product classifying algorithm - text classification - C# - algorithm suggestions

Alright people. Finally with the help of stackoverflow community i have gathered 20 commercial product selling websites product pages with the following features Product URL Product Price Product ...
3 votes
2 answers
647 views

Available options for classifying words in text?

I am researching ways to classify words in text and I'm wondering what options there are and which are best suited to this job. I'm mostly interested in keywords which are most often nouns. So far I ...
  • 1,203
2 votes
3 answers
325 views

Find the least words that will use all given letters

With a list of thousands of words and a small list of letters I am trying to find the least amount of words to make use of all given letters, assuming my dictionary of words covers all letters. The ...
  • 131
2 votes
2 answers
2k views

Hardware accelerated text processing

Graphics processing units (GPUs) are very common and allow for efficient, parallel processing of floating point numbers. PPUs (Physics Processing Units) used to be a buzzword several years ago but ...
  • 1,345
2 votes
2 answers
9k views

Fastest C++ XML parsing library

I have thousands of .xml files from size 1MB-45MB (no DTDs). I need to parse and further manipulate these XML files before generating separate .xml files with the results of my regex. What the ...
  • 727
2 votes
1 answer
601 views

Tokenizing Text Held in a Rope Data Structure

I am building a text editor which makes use of a Ragel based tokenizer to support syntax highlighting. I am considering the use of a rope data structure to support efficient modifications and undo/...
  • 75
2 votes
3 answers
2k views

Best algorithm to correlate similar articles [closed]

which is the best way to correlate and group similar articles? I mean something like Google News, which groups under a single topic different articles from different sources. I'm not interested in ...
  • 131
2 votes
0 answers
48 views

How to test generated text

I am creating a text generation algorithm for my master's research. I have a dialogue between two people and I would like to simulate one part of the conversation with naturally generated text (not ...
1 vote
1 answer
3k views

How to load text files into memory-mapped files

I have a number of large text files that I need to manipulate in a highly performant manner. I've decided to look into using Memory Mapped files in C# (.NET 4). However, I can't find any examples of ...
  • 801
1 vote
2 answers
564 views

Inserting copyright notice [closed]

What is the easiest way to insert copyright notice in lots of PHP files. It's not possible to do it manually.
  • 313
1 vote
1 answer
83 views

When to have a generic method for multiple tasks when some differ?

I process a lot of tweets in real time using python and for each tweet I need to assign it in to a specific bucket. I have about 50 buckets, each with their own rules. The majority of them are simple ...
  • 113
1 vote
1 answer
85 views

How to analyse user input and determine if it (kind of) matches an answer

I'm working on a quiz system that will allow users to enter text as an answer. The question could be something simple to start with, looking for a short phrase, or a select few words as the "correct" ...
  • 133
1 vote
1 answer
1k views

An elegant way to split text into words combined with adjacent punctuation and determine which punctuation mark it is

Firstly, I realize that question title is about as terrible as the sample code I'll post below, so please bear with me while I explain the problem more clearly, and if you have a better idea for the ...
1 vote
4 answers
188 views

Possible applications of algorithm devised for differentiating between structured vs random text

I have written a program that can rapidly (within 5 sec on a 2GB RAM desktop, 2.33 Ghz CPU) differentiate between structured text (e.g English text) and random alphanumeric strings. It can also ...
  • 11
1 vote
2 answers
324 views

Automatic summarization - whole text or paragraphs

I am writing a tool that will give users the ability to summarize text content on a webpage, by highlighting the text that they wish to get summarized. So far, I've received results that I can work ...
  • 111
1 vote
2 answers
688 views

Sorting Sentences by New Words in Each

A very useful learning tool I stumbled across for Chinese was a massive list of sentences that, barring the first 10 or 15, only differed by the ones before by one or two words, or at least as few as ...
1 vote
1 answer
1k views

Custom Alphabetic Sorting of Array in Java

I have a requirement to read a text file with lines in tag=value format and then output the file with specific tags listed first and the rest sorted alphabetically. The incoming file is randomly ...
1 vote
0 answers
113 views

What are the methods to accurately test the accuracy of a parts of speech tagger?

I know that most of the pos tagger algorithms measure their accuracy token wise I.e. whether the token is tagged correctly or not Some pos taggers provide sentence accuracy too. How is sentence ...
  • 169
0 votes
1 answer
353 views

Why are ordinal indicators considered letters?

It got my by surprise yesterday than ordinal indicators are considered letters. I thought letters were only [a-zA-Z]. Why are they considered letters and not symbols? char.IsLetter('º'); // true ...
0 votes
2 answers
12k views

Name and code to space between lines/paragraphs

I’m seeking a term and possibly the code behind what would help me implement that term in Python. I have been working on a text-based Python journaling application. When I want to review my ...
0 votes
1 answer
83 views

How to implement tracking of changes in text documents à la MS-Word/Apple Pages

I want to implement tracking of changes in plain-text documents, in a way similar to how it works in MS Word or Apple Pages. What I am unsure of is the data model and how to store it. Goal The ...
  • 2,037
0 votes
2 answers
269 views

Optimizing sorting large amounts of text stored in a database

How would you store text information - in a relational database or maybe using NoSQL? The problem is that the text should be divided into various parts, each of which satisfies to some requirement, e....
  • 1,381
0 votes
1 answer
89 views

Database of big text documents many-to-many: one big relationship table, a lot of small ones, or a better way to link abstract text data?

so I am struggling a bit with a database setup. I found post with similar problems, but the reason behind the answers was not what I was looking for, hence I ask again with my specifics. I am building ...
  • 31
0 votes
1 answer
111 views

Convert RTF to HTML when it's saved to the database or when it's rendered?

Users have the ability to enter and save text in a rich text editor which is eventually stored in a database and then rendered on a site. Is it better to convert the RTF to HTML when it's stored to ...
  • 190
0 votes
1 answer
187 views

Algorithm for line breaking in monospace text

Is there a de facto standard algorithm for finding good places to put line breaks in a paragraph of text rendered in a monospace font (e.g. to a text console)? The algorithm should aim to output lines ...
  • 125
0 votes
2 answers
1k views

How does Facebook strip html/apostrophes for XSS but also display it?

I'm not quite sure if this is a question for programmers.se rather than stackoverflow, but here goes. So Facebook [or any other large company] when given something like an apostrophe or html, can ...
  • 191
0 votes
1 answer
102 views

Is my understanding of modern IDEs autosave feature naive? [duplicate]

Long time ago I learned that text files are not like Random access Files, i. e., adding or updating info at the beginning of a text file involves moving all the rest of the file "forward" (or ...
  • 1
0 votes
0 answers
48 views

Synchronizing webpage and local pdf

I would like to solve the following problem. On my website, I have a list of my publications. I also have my list of publications on a latex file of my cv. The issue is that I update these manually, ...
-1 votes
3 answers
125 views

Is there a text distance (or string similarity) algorithm which accounts for the distance between characters?

I'm interested in finding a text distance (or string similarity) algorithm which computes a greater distance (or lower similarity) when characters are further apart. For example, I want the distance ...