Page History

...

(4) Diving deeper into text.  Optical Character Recognition (OCR) processes are not perfect. Consider some areas of special challenge:

...

Computational analysis of text

We count tokens - What is tokenization?  Why tokenize?  What are some strategies used to tokenize?

(1) Let's look again at the Arapaho gospel of St. Luke.  Switch to text view.

• Is this OCR accurate to the visually captured page? 
• What is a word? How would you define this for "word" to a computer?
• What isn't a word? How would you tell a computer to exclude these?
• Consider languages with which you are familiar.  Can you think of special cases where tokens might contain more than one word?
• What sets of rules would we need in order to tokenize effectively?  Would these be ordered in any specific way?
• Is there a "right way" to tokenize?

(2) Discussion: What are the opportunity points that the structure and arrangement of a book afford?

...

• Sample texts, courtesy of Project Gutenberg. Use the plain text version.
  
  • Crane, Stephen, 1871-1900. The Red Badge of Courage: An Episode of the American Civil War.
  • Dickens, Charles, 1812-1870. A Tale of Two Cities.
  • Upham, Charles Wentworth, 1802-1875. Salem Witchcraft, Volumes I and II
• Sample URLS: copy and paste into the Voyant upload browser window to get started.Sample URLS: copy and paste into the Voyant upload browser window to get started.

  • Economics of Crisis - http://www.economicsofcrisis.com/indications.html 

  • Instructions to major John Sullivan. Washington, George, 1732-1799. The writings of George Washington from the original manuscript sources. Electronic Text Center, University of Virginia LibraryInternet Archive.

  • Copyright Law of the United States of America and Related Laws Contained in Title 17 of the United States Code - http://www.copyright.gov/title17/92preface.html
• Sample visualization: Dr. Martin Luther King, Jr. I Have a Dream speech.

...

• Explore visualizations in the "dashboard" that results from analysis of uploaded text.  Change Explore changing the options for visualizations. 
• Discuss the relative merits of the various visualizations. 

...

• We calculate frequency
  

  • We can express our counts simply (as counts), or we can express them as frequencies.  Why calculate frequencies?
  • Is either representation misleading?  If so, in what ways?
• What does exerting control do to our results?  Does it change the validity of our assertions?
• How should method be explained when making assertions from results?
• Who determines what is "signal" and what is "noise"What is signal?  What is noise?

Moving from Microanalysis to Macroanalysis (Google nGrams and HTRC bookworm)

nGrams are words or phrases, tokenized and counted in a defined corpus and displayed as a graph showing how relative frequencies of those phrases have occurred as occurring over publication date. The two tools referenced below provide a basis for exploration of ngrams.  Each tool is bound to secondary data derived from analysis of a different corpus, so results of the same nGram will not necessarily align. 

...

• How easily can these examples above be refined and improved?
• Compare the two interfaces, especially as to the affordances and the limits of each. 
• What additional elements of control would be useful that aren't available?
• When we see unexpected wave forms, what do we make of these? 
  
  • Do these constitute discoveries or represent errors?  How can we distinguish?
  • Would the flaws be due to the data, the metadata, the algorithms?

...

• Compare one collection of books to a second collection, and report the differences in frequency of tokens - ShksprDunning
• Extract entities from a set of books, and list out referents of where they occur
  
  • Person, location - WSPlaysEntityExtract
  • Dates - WSComediesExtractDates (Note the wide variety of what is considered a "date")
  • Dates visualized over timeline - WSComediesDateExtractSimileRemix
• Model "topics", or clusters of tokens that are statistically more likely to be found together - WSComediesTopics; WSTragediesTopics

Discussion

• In general, do the results look valid? Do any of these algorithms yield results that might be considered confusing or less than perfect?

• Note that results can be downloaded.  What might be advantages of this portability?Note the portability of the results.  Can you think of advantages to this? 
• Are there things that the researcher would want or need to know about these algorithms when making claims about results?

...