Page History

...

(1) Let's look again at the Arapaho gospel of St. Luke.  Switch to text view.

• Is this OCR accurate to the visually captured page? 
• What is a word? How would you define "word" to a computer?
• What isn't a word? How would you tell a computer to exclude these?
• Consider languages with which you are familiar.  Can you think of cases where tokens might contain more than one word?
• What sets of rules would we need in order to tokenize effectively?  Would these be ordered in any specific way?
• Is there a "right way" to tokenize?

...

nGrams are words or phrases, tokenized and counted in a defined corpus and displayed as a graph showing how relative frequencies of those phrases have occurred as occurring over publication date. The two tools referenced below provide a basis for exploration of ngrams.  Each tool is bound to secondary data derived from analysis of a different corpus, so results of the same nGram will not necessarily align. 

...

• How easily can these examples above be refined and improved?
• Compare the two interfaces, especially as to the affordances and the limits of each. 
• What additional elements of control would be useful that aren't available?
• When we see unexpected wave forms, what do we make of these? 
  
  • Do these constitute discoveries or represent errors?  How can we distinguish?
  • Would the flaws be due to the data, the metadata, the algorithms?

...

• Compare one collection of books to a second collection, and report the differences in frequency of tokens - ShksprDunning
• Extract entities from a set of books, and list out referents of where they occur
  
  • Person, location - WSPlaysEntityExtract
  • Dates - WSComediesExtractDates (Note the wide variety of what is considered a "date")
  • Dates visualized over timeline - WSComediesDateExtractSimileRemix
• Model "topics", or clusters of tokens that are statistically more likely to be found together - WSComediesTopics; WSTragediesTopics

Discussion

• In general, do the results look valid? Do any of these algorithms yield results that might be considered confusing or less than perfect?

• Note that results can be downloaded.  What might be advantages of this portability?Note the portability of the results.  Can you think of advantages to this? 
• Are there things that the researcher would want or need to know about these algorithms when making claims about results?

...