I would like to store the frequencies with which words co-occur with each other over a variety of contexts in a large (> 1 billion tokens) text corpus. I need to store the word pair, the type of co-occurrence (e.g. word1 in the same sentence as word2, word1 in the same paragraph as word2), and some metadata about the text in which the co-occurrence was found, (e.g. year, author, publisher). So a single row might look like:
word1 word2 count decade publisher author context_type
---------------------------------------------------------------------
nuclear danger 22 1980s NYT Mary Smith paragraph
The frequencies will be sparse and Poisson distributed. I would then like to be able to query the data by date ranges, or groups of authors (for example), aggregating the counts of the results.
I have little experience with databases and am not sure what to use. Do I need related SQL tables (e.g. with the book metadata in one table and word data in another), or a simple flat NoSql solution? The vocabulary is about 50,000 words, so if every word-word co-occurrence was observed there would be 2.5 billion rows even without the metadata (I think). But probably most of them won't be observed. This makes me think maybe a graph database is a possible solution. Is there a good cloud solution on AWS or google maybe?