DSPS Summer Fellows 2016 - Computational Analysis

An overview of computational analysis, and exploration of strategies and challenges.

Prep

• bring text to play with in Voyant

Anatomy and Physiology of a Digital Book

• Two books - page images and companion OCR - (sample book from HTDL)
  • What our eyes like to read is images
  • What machines read  needs to be constrained and predictable:"machine actionable"
• Books have metadata -
  
  • bibliographic (catalog view, MARC view)
  • structural
• OCR - where does it come from?
• OCR issues
• • language specific challenges
    
    • Roman/non Roman
    • Special character sets: Japanese done well, and not so well
    • Mixed languages: Latin/Greek 
  • font styles
  • image resolution
  • handwriting

We count words - tokenization; why tokenize?

• What is a word? (spaces are like any other character for a machine)
• What isn't a word? 
  
  • Punctuation
  • page numbers
  • grammatical constructs of various languages (ex: negation, especially in French; passive reflexive in Spanish)
• What words do we prefer not to count?
  
  • headers/footers

Introducing Control - "Microanalysis" and Voyant

• Voyant
• Load your text sample
• Playing with control
  • observe counts
  • stopwords
• What does exerting control do to our results?  Does it change the validity of our assertions?
• What is signal?  What is noise?

We calculate frequency

• Why not express our counts simply?  Why calculate frequencies?
• Is this misleading?  If so in what ways?

Moving from Microanalysis to Macroanalysis

• Google nGrams; help (examples)
• Bookworm; help, (examples)
• problems of convergence/divergence and strategies for disentangling.

More macroanalysis

• HTRC
• entity extraction
  • What are entities?
  • False positives/false negatives (omissions)
• Clustering - topic modeling
  • define and defer for Mimno

Image analysis

• Edge analysis
• Neural networks

Biblio

Writing about your results - the structure of scientific papers

Matt Jockers Book