Tooling Up for Digital Humanities

As archives increasingly go digital, what do humanities scholars need to know in order to optimize their use of these new resources?

First and foremost, the format of those resources matters. A digital archive represents the work and decisions of many librarians and archivists, and understanding those decisions and the forms they take can help researchers access and manipulate data more effectively. Certain types of digital analysis can only be done on data that exists in certain formats, for instance. Although scholars need not understand the digital archiving process in enormous depth, they must understand on a basic level how a particular online archive is structured.

The first step in the archival process of transforming a source from an analog (non-digital) format to an electronic format usually takes the form of scanning the document in order to create a digitized image of the text (a PDF, for instance). This initial imprint is like taking a photograph, even if the imprint is of a page of text.  A great deal of archival material is available in this “images of words” format. Although useful for someone who wants to read read the document, this image alone does little to help researchers find, access, or manipulate that text.

Documents become more usable through a process called “extraction,” in which computer software performs Optical Character Recognition (OCR). This process scans the image of text and attempts to recognize characters and words, which it then stores as a separate layer of text based on the scanned image. This layer can be thought of as being superimposed on the image – while the human eye understands a picture of the word “Uzbekistan,” the computer uses the translated, invisible layer to recognize “Uzbekistan” as a series of characters that form a word.

This process is largely automated, although the software can be checked and supplemented by a human reader. A fully-automated process is commonly known as a “dirty OCR” because it will often have many typos and errors (letter combinations like “rn” might be transcribed as “m,” for instance.). The initial OCR text translation is usually around 70 or 80 percent accurate, and often needs human review. It is important to recognize that when performing searches within OCR’d documents the search is not necessarily comprehensive. If a word was not properly recognized or translated by the process, a computer search will not find it (even if a human reader will correctly recognize the word). From here, the image and “dirty OCR” can be reviewed by a human reader to improve its translation accuracy.