*** This version of Confluence is for testing only and contains a copy of content from June 29th 2026. No changes will be preserved. ***
...
(4) Diving deeper into text. Optical Character Recognition (OCR) processes are not perfect. Consider some areas of special challenge:
...
Computational analysis of text
We count tokens - What is tokenization? Why tokenize? What are some strategies used to tokenize?
(1) Let's look again at the Arapaho gospel of St. Luke. Switch to text view.
- What is a word? How would you define this for "word" to a computer?
- What isn't a word? How would you tell a computer to exclude these?
- Consider languages with which you are familiar. Can you think of special cases where tokens might contain more than one word?
- What sets of rules would we need in order to tokenize effectively? Would these be ordered in any specific way?
- Is there a "right way" to tokenize?
(2) Discussion: What are the opportunity points that the structure and arrangement of a book afford?
...
- Sample texts, courtesy of Project Gutenberg. Use the plain text version.
- Crane, Stephen, 1871-1900. The Red Badge of Courage: An Episode of the American Civil War.
- Dickens, Charles, 1812-1870. A Tale of Two Cities.
- Upham, Charles Wentworth, 1802-1875. Salem Witchcraft, Volumes I and II
- Sample URLS: copy and paste into the Voyant upload browser window to get started.Sample URLS: copy and paste into the Voyant upload browser window to get started.
Economics of Crisis - http://www.economicsofcrisis.com/indications.html
Instructions to major John Sullivan. Washington, George, 1732-1799. The writings of George Washington from the original manuscript sources. Electronic Text Center, University of Virginia Library. Internet Archive.
- Copyright Law of the United States of America and Related Laws Contained in Title 17 of the United States Code - http://www.copyright.gov/title17/92preface.html
- Sample visualization: Dr. Martin Luther King, Jr. I Have a Dream speech.
...
- Explore visualizations in the "dashboard" that results from analysis of uploaded text. Change Explore changing the options for visualizations.
- Discuss the relative merits of the various visualizations.
...
We calculate frequency
- We can express our counts simply (as counts), or we can express them as frequencies. Why calculate frequencies?
- Is either representation misleading? If so, in what ways?
- What does exerting control do to our results? Does it change the validity of our assertions?
- How should method be explained when making assertions from results?
- Who determines what is "signal" and what is "noise"What is signal? What is noise?
Moving from Microanalysis to Macroanalysis (Google nGrams and HTRC bookworm)
...