*** This version of Confluence is for testing only and contains a copy of content from June 29th 2026. No changes will be preserved. ***
...
- Bring text to play with in Voyant. Input format can be plain text, a PDF (with OCR), a MS Word Document or a URL for HTML analysis. Upload of any material will be subject to the Voyant privacy policy, so bring text you can safely share.
- Bring a laptop to support your explorations.
- Optionally - the intrepid may want to obtain a login to the HTRC Portal, create a workset and run a few algorithms in advance of this lecture. Documentation for obtaining a sign-on and documentation for the portal is helpful.
...
(2) Now explore the controls at he the top of the page turner.
- There is a box labeled "search in this text". What can you deduce about the book from this functionality?
- What do the other controls do? Is there a way to summarize this class of controls? What underlying logic might you predict that coordinates these functions? (Food for thought...)
...
We count tokens - What is tokenization? Why tokenize? What are some strategies used to tokenize?
(1) Let's look again at the Arapaho gospel of St. Luke. Switch to text view.
- What is a word? How would you define this for a computer?
- What isn't a word? How would you tell a computer to exclude these?
- Can you think of special cases where tokens might contain more than one word?
- What sets of rules would we need to tokenize? Would these be ordered in any specific way?
- Is there a "right way" to tokenize?
(2) Discussion What are the opportunity points that the structure and arrangement of a book afford?
- How do challenges with OCR intersect with strategies for computational analysis of text? What might be effective strategies to deal with these challenges?
- What exactly is the "text"? Can you think of parts of a book that you might not want to include in your analysis? Why or why not? If you would, how would you exclude these parts?
...
- stopwords
- slider for word counts
- counts vs frequencies
(3) General questionsDiscussion
- What does exerting control do to our results? Does it change the validity of our assertions?
- What is signal? What is noise?
...
- monster in fiction/non-fiction/unknown
- mystery (contrast fiction and nonfiction)
- various words for "creole"
- "phonograph" in three languages
General questions for discussionDiscussion
- How easily can these examples above be refined and improved?
- Compare the various interfaces, especially as to the affordances and the limits of each.
- What additional elements of control would be useful that aren't available?
- When we see unexpected wave forms, what do we make of these?
- Do these constitute discoveries or errors? How can we distinguish?
- Would the flaws be due to the data, the metadata, the algorithms?
More Macroanalysis
...
: HTRC
HathiTrust Research Center (HTRC) is a collaborative research center (jointly managed by Indiana University and the University of Illinois) dedicated to developing cutting-edge software tools and cyberinfrastructure that enable advanced computational access to large amounts of digital text. A basic orientation of HTRC services is available, and features and steps for each are documented on the HTRC community wiki. We will be spending time in the HTRC Portal looking at the results of a few algorithms as a sampling of possibilities.
...
- What are entities?
- False positives/false negatives (omissions)
Agorithms can
- Compare one collection of books to a second collection, and report the differences in frequency of tokens - ShksprDunning
- Extract entities from a set of books, and list out referents of where they occur
- Person, location - WSPlaysEntityExtract
- Dates - WSComediesExtractDates
- Dates over timeline - WSComediesDateExtractSimileRemix
- Model "topics", or clusters of tokens that are statistically more likely to be found together - WSComediesTopics; WSTragediesTopics
Discussion
- Are there cases where
...
Image analysis
Ukiyo-e.org is a database and image similarity analysis engine, created by John Resig to aide researchers in the study of Japanese woodblock prints. The data is over 213,000 digital copies of prints from 24 institutions, and their cataloging metadata. Metadata is indexed and searchable. (Details are noted in the about page.) Resig'sImage search uses the TinEye matching engine to determine edges in an uploaded sample and compares with analyzed edges in database, returning probable matches (edge analysis).
...