*** This version of Confluence is for testing only and contains a copy of content from June 29th 2026. No changes will be preserved. ***
...
- Two books - page images and companion OCR - (sample book from HTDL)
- What our eyes like to read is images
- What machines read needs to be constrained and predictable.:"machine actionable"
- Books have metadata -
- bibliographic (catalog view, MARC view)
- structural
- OCR - where does it come from?
- OCR issues
- language specific dictionaries
- font styles
- image resolution
- handwriting
We count words - tokenization; why tokenize?
- What is a word? (spaces are like any other character for a machine)
- What isn't a word?
- Punctuation
- page numbers
- grammatical constructs of various languages (ex: negation, especially in French; passive reflexive in Spanish)
- What words do we prefer not to count?
- headers/footers
...
- Voyant
- Load your text sample
- Playing with control
- observe counts
- stopwords
- What does exerting control do to our results. ? Does it change the validity of our assertions?
- What is signal? What is noise?
We calculate frequency
- Why not express our counts simply? Why calculate frequencies?
- Is this misleading? If so in what ways?
Biblio
Writing about your results - the structure of scientific papers
Matt Jockers Book