Page History

...

Computational analysis of text

We count tokens - What is tokenization?  Why tokenize?  What are some strategies used to tokenize?

(1) Let's look again at the Arapaho gospel of St. Luke.  Switch to text view.

...

• Sample texts, courtesy of Project Gutenberg. Use the plain text version.
  
  • Crane, Stephen, 1871-1900. The Red Badge of Courage: An Episode of the American Civil War.
  • Dickens, Charles, 1812-1870. A Tale of Two Cities.
  • Upham, Charles Wentworth, 1802-1875. Salem Witchcraft, Volumes I and II
• Sample URLS: copy and paste into the Voyant upload browser window to get started.Sample URLS: copy and paste into the Voyant upload browser window to get started.

  • Economics of Crisis - http://www.economicsofcrisis.com/indications.html 

  • Instructions to major John Sullivan. Washington, George, 1732-1799. The writings of George Washington from the original manuscript sources. Electronic Text Center, University of Virginia Library

  • Copyright Law of the United States of America and Related Laws Contained in Title 17 of the United States Code - http://www.copyright.gov/title17/92preface.html
• Sample visualization: Dr. Martin Luther King, Jr. I Have a Dream speech.

We calculate frequency

• We can express our counts simply (as counts), or we can express them as frequencies.  Why calculate frequencies?
• Is either representation misleading?  If so, in what ways?

(1) Visualization of derived data

...

• stopwords
• slider for word counts
• counts vs frequencies

(3) Discussion

• We calculate frequency. 
  

  • We can express our counts simply (as counts), or we can express them as frequencies.  Why calculate frequencies?
  • Is either representation misleading?  If so, in what ways?
• What does exerting control do to our results?  Does it change the validity of our assertions?
• What is signal?  What is noise?

...

• How easily can these examples above be refined and improved?
• Compare the various two interfaces, especially as to the affordances and the limits of each. 
• What additional elements of control would be useful that aren't available?
• When we see unexpected wave forms, what do we make of these? 
  
  • Do these constitute discoveries or errors?  How can we distinguish?
  • Would the flaws be due to the data, the metadata, the algorithms?

More Macroanalysis: HTRC

HathiTrust Research Center (HTRC) is a collaborative research center (jointly managed by Indiana University and the University of Illinois) dedicated to developing cutting-edge software tools and cyberinfrastructure that enable advanced computational access to large amounts of digital text. A basic orientation of HTRC services is available, and features and steps for each are documented on the HTRC community wiki.  We will be spending time in the HTRC Portal looking at the results of a few algorithms as a sampling of possibilities .  Agorithms (links below will not render for all, but are parked to make sharing easier).  Algorithms in the portal can

• Compare one collection of books to a second collection, and report the differences in frequency of tokens - ShksprDunning
• Extract entities from a set of books, and list out referents of where they occur
  
  • Person, location - WSPlaysEntityExtract
  • Dates - WSComediesExtractDates
  • Dates over timeline - WSComediesDateExtractSimileRemix
• Model "topics", or clusters of tokens that are statistically more likely to be found together - WSComediesTopics; WSTragediesTopics

Discussion

• Are there cases where

Image analysis

Ukiyo-e.org is a database and image similarity analysis engine, created by John Resig to aide researchers in the study of Japanese woodblock prints.  The data is over 213,000 digital copies of prints from 24 institutions, and their cataloging metadata.  Metadata is indexed and searchable. (Details are noted in the about page.)  Resig'sImage search uses the TinEye matching engine to determine edges in an uploaded sample and compares with analyzed edges in database, returning probable matches (edge analysis).

...