Page History

...

Computational analysis of text

We count tokens - What is tokenization?  Why tokenize?  What are some strategies used to tokenize?

...

Voyant is a low barrier text analysis tool that delivers a rich, interactive interface and a variety of visualizations based on token counts within a single or a few texts.  Input format can be plain text, a PDF (with OCR), a an MS Word Document or a URL for HTML analysis. Upload of any material will be subject to the Voyant privacy policy. Sample texts and URLs for analysis are listed below for experimentation, but feel free to use other source data that interests you. 

...

(1) Visualization of derived data

• Explore visualizations in the "dashboard".  Change the options in the dashboardfor visualizations. 
• Discuss the relative merits of the various visualizations. 

(2) Exerting control

• Experiment with stopwords
• Experiment with the slider for word counts
• counts Consider raw vs relative frequencies

(3) Discussion

• We calculate frequency. 
  

  • We can express our counts simply (as counts), or we can express them as frequencies.  Why calculate frequencies?
  • Is either representation misleading?  If so, in what ways?
• What does exerting control do to our results?  Does it change the validity of our assertions?
• What is signal?  What is noise?

Moving from Microanalysis to Macroanalysis (Google nGrams and HTRC bookworm)

nGrams are words or phrases, counted in a defined corpus and displayed as a graph showing how those phrases have occurred over publication date. The two tools referenced below provide a basis for exploration of ngrams.  Each tool is bound to secondary data derived from analysis of a different corpus, so results of the same nGram will not necessarily align. 

Google's nGram Viewer. Use the links below as starting points; dynamic modifications can be made at any point. Rules for syntax can be found on the About page.

...

• Compare one collection of books to a second collection, and report the differences in frequency of tokens - ShksprDunning
• Extract entities from a set of books, and list out referents of where they occur
  
  • Person, location - WSPlaysEntityExtract
  • Dates - WSComediesExtractDates
  • Dates over timeline - WSComediesDateExtractSimileRemix
• Model "topics", or clusters of tokens that are statistically more likely to be found together - WSComediesTopics; WSTragediesTopics

Discussion

• Note the portability of the results.  Can you think of advantages to this? 
• Are there things that the researcher would want or need to know about these algorithms when making claims about results?

Image analysis

Ukiyo-e.org is a database and image similarity analysis engine, created by John Resig to aide researchers in the study of Japanese woodblock prints.  The data is over 213,000 digital copies of prints from 24 institutions, and their cataloging metadata.  Metadata is indexed and searchable. (Details are noted in the about page.)  Resig'sImage search uses the TinEye matching engine to determine edges in an uploaded sample and compares with analyzed edges in database, returning probable matches (edge analysis).

...