13

I am trying to gather statistics on character or word sequences used in the English language for use in a software project.

Where can I can get a large amount (several GB would be nice) of English plain-text covering a diverse set of topics?

0

4 Answers 4

18

You can use Wikipedia's data dumps. The XML data dump for English Wikipedia that includes current revisions only is about 31 GB, so I'd say it would be a good start for your research. The data dump is pretty big, so you should consider extracting the texts from XML with a SAX parser. WikiXMLJ is a handy Java API tuned for Wikipedia.

And then, of course, there is always the Stack Exchange data dumps. The latest one includes all public non-beta Stack Exchange sites & corresponding Meta sites up until September 2011. But, naturally Stack Exchange posts are concentrated on the scope of each site, so probably not as generalized as you'd wish. Meta posts are a bit more general though, so you could consider those in addition to Wikipedia.

I don't think you'll find anything better, especially in plain text. Several open data sets are available through the Data Hub, but I think the English Wikipedia data dump is very close to what you are looking for.

1
  • The Stack ones, while extensive, are going to cover a very narrow field of discourse (by necessity), so they may not generalize well.
    – jonsca
    Feb 1, 2012 at 1:46
5

Google has a collection of data sets that they use to determine n-gram probabilities. Examining their bigram (2-gram) datasets should give you a good picture. There are many other corpi out there for which these analyses have already been done.

0
5

Project Gutenberg has a large corpus of texts in English, already in text form.

Project Gutenberg offers over 42,000 free ebooks: choose among free epub books, free kindle books, download them or read them online.

We carry high quality ebooks: All our ebooks were previously published by bona fide publishers. We digitized and diligently proofread them with the help of thousands of volunteers...

4
  • 1
    I thought about Project Gutenberg but I couldn't find a concentrated data dump. And for a book to be included, it's copyright must expire, and generally that means that 50 to 70 years have passed since the books first publication. So I don't think that as a data set, Project Gutenberg is representative of the language as used today.
    – yannis
    Feb 1, 2012 at 1:59
  • 1
    If you want something that is "representative of the language as used today", try YouTube comments. Sad but true. Feb 1, 2012 at 3:49
  • @JörgWMittag - ouch. What really bothers me is how not wrong you are. Feb 1, 2012 at 12:12
  • @Jörg W Mittag It's possible, but then certain words specific to youtube would come up very frequently, like: YO OU UT TU UB BE, or even worse: FA AK KE AN ND GA AY
    – JSideris
    Feb 1, 2012 at 20:06
1

For the statistics, you are probably looking at "Bigram Frequency in the English language". Take a look at: Wiki-Bigram Stats

as for finding a large text, note that the frequency would be biased to the type of text. For example, if you analyze addresses you will get different results from analyzing newspaper stories. If you just want to test, you could use any book's PDF file (better not be a math. or programming or medical book) and convert it to text then run your tests. You could also convert newspaper web pages to text and work on those.

1
  • 2
    Yea I realize that the results are going to be biased. I need a resource that covers as many subjects as possible. I considered downloading a bunch of e-books, main problem is converting them all to text. But it wouldn't hurt to look up some bigram statistics (I didn't realize that's what 2-letter combinations were called).
    – JSideris
    Feb 1, 2012 at 1:45

Not the answer you're looking for? Browse other questions tagged or ask your own question.