Page History

An overview of computational analysis, and exploration of strategies and challenges.

Prep

• bring text to play with in Voyant

Going down a rabbit hole: anatomy of a digital book

How is a digital book made?  How does the structure relate to it's functionality? What opportunities does this afford us in terms of text analysis?

...

• There is a box labeled "search in this text".  What can you deduce about the book from this functionality? 
• The arrows move the view back and forth in the book. The sections allow for jumping to chapter starts.  What must enable this function? (Food for thought...)

What might this page be?  (It also has this view.) Is this also part of the book? 

Diving deeper into plaintext: OCR processes are not perfect.

Consider some problem areas:

• poor image quality (resolution, warp, skew, crop)
• language specific challenges
• font styleshandwriting
• Unexpected arrangement
• Roman/non Roman
• Special character sets: Japanese done well, and not so well
• Mixed languages: Latin/Greek 

...

• What is a word? (spaces are like any other character for a machine)
• What isn't a word? 
  
  • Punctuation
  • page numbers
  • grammatical constructs of various languages (ex: negation, especially in French; passive reflexive in Spanish)
• What words do we prefer not to count?
  
  • headers/footers
  • preface
  • indexes
  • title pages
  • abstracts

Introducing Control - "Microanalysis" and Voyant

• Voyant
• Load your text sample
• Playing with control
  • observe counts
  • stopwords
• What does exerting control do to our results?  Does it change the validity of our assertions?
• What is signal?  What is noise?

...

• Why not express our counts simply?  Why calculate frequencies?
• Is this misleading?  If so in what ways?

Moving from Microanalysis to Macroanalysis

• Google nGrams; help (examples)
• Bookworm; help, (examples)
• problems of convergence/divergence and strategies for disentangling.

More macroanalysis

• HTRC
• entity extraction
  • What are entities?
  • False positives/false negatives (omissions)
• Clustering - topic modeling
  • define and defer for Mimno

Image analysis

• Edge analysis
• Neural networks

Biblio

Writing about your results - the structure of scientific papers

...