Natural Language Processing for Historical Texts

Transkript

1 Natural Language Processing for Historical Texts GSCL 2013 Tutorial Dr-Ing Michael mxp Leibniz Institute of European History September 24, 2013 Piotrowski September 24, 2013 NLPHist 1/1 Acquiring historical texts Rarely discussed for modern texts: Modern texts are easy to obtain in electronic form The bulk of historical text has not yet been digitized For regular NLP research, text acquisition is usually not an issue Typical workflow for historical texts: 1 Acquire text in its original form (eg, as a book) 2 Scan it 3 Convert images to digital text Piotrowski September 24, 2013 NLPHist 3/1

2 Decision criteria If the texts have already been digitized (eg, in a previous project or for Project Gutenberg), error checking and correction may be required to achieve the desired quality If the texts are available in printed form with good print quality and in roman type, optical character recognition (OCR) or double-keying may be options; for both approaches the documents have to be scanned first If the texts are printed, and the print quality is low or the texts are printed in blackletter types, it depends on the language and the desired quality whether OCR (with post-processing) is still an option, otherwise qualified double-keying is likely to be required If the texts are handwritten, double-keying by qualified personnel is currently the only option (even though there is research on automatic handwriting recognition for historical documents) Piotrowski September 24, 2013 NLPHist 4/1 Scanning Scanning makes sense even if the text will be keyed in Focus on printed books Can books be cut open? Sheet-fed scanners Otherwise: Flat-bed or book scanners with automatic or manual page turning In-house or contractor? Piotrowski September 24, 2013 NLPHist 6/1

3 Notes on scanners Piotrowski September 24, 2013 NLPHist 7/1 Notes on scanners When scanning in-house, select the right scanner for the job If you have 50,000 or 100,000 pages to scan, speed and reliability become important Example: If 65,000 pages are to be scanned with a scanner that scans 21 pages per minute, total scanning time is pages 21 pages/min = min = 516 h This corresponds to about seven eight-hour days, but additional time is needed for loading the ADF, separating pages that are sticking together, etc When comparing scanners, note which scan parameters (resolution and depth) were used for speed measurements Daily or monthly duty cycle gives an indication of mechanical quality and suitability for the job size Piotrowski September 24, 2013 NLPHist 8/1

4 Scan parameters Optimal scan parameters depend on intended use In general: Higher scan resultion is better, as it captures more details Lower-resolution images can easily be created by scaling down the inverse is not possible Resolution must actually be provided by scanner (optical), not interpolated Higher scan resolution means slower scanning In practice: Books usually scanned at 300 or 600 dpi Piotrowski September 24, 2013 NLPHist 9/1 Bilevel or grayscale? Should text intended for OCR be scanned in black and white or grayscale? Actual OCR is always done on bilevel images OCR software vendors typically recommend grayscale scanning: Want to use their own binarization algorithms But: Not certain whether it actually improves OCR quality Holley (2009): No significant improvement Powell (2009) Plus: Processing of grayscale files increases cost of OCR: Larger files are slower to process for OCR software and need more storage space 100,000 bilevel images, 600 dpi, Group 4: 83 GB 100,000 grayscale images, 600 dpi, Zip: 3613 GB Not a storage problem today, but backup, throughput and general file management are still issues Run experiments with test pages from the actual documents If grayscale does not yield significant improvements, use higher-resolution bilevel scans Piotrowski September 24, 2013 NLPHist 10/1

5 File sizes Example: Scans of typical book pages Resolution Depth Compression File size/page (dpi) (bpp) (MB) Group Group none Zip Group Piotrowski September 24, 2013 NLPHist 11/1 File formats Preferred format for bilevel data: TIFF with Group 4 compression Even better compression: JBIG2, but encoding not yet widely available 50 scanned pages at 600 dpi ( pixels): Group 4: 56 MB JBIG2: 744 KB Group 4 and JBIG2 data can be embedded in PDF files: No special viewer required Commercial solution: CVISION PDFCompressor, Open source: Sojka (2010) For grayscale: TIFF with Zip compression or PNG JPEG is lossy and designed for photographs Never use JPEG for text! Piotrowski September 24, 2013 NLPHist 12/1

6 OCR High quality of OCR output on modern documents critically depends on NLP resources, in particular dictionaries If pattern recognition engine cannot decide between multiple possible readings (eg, cunent vs current) dictionary is consulted Readings in dictionary will be preferred Additional language-specific rules help resolve issues such as capitalization and punctuation (eg, commas vs periods, and lowercase ell vs capital I vs number 1) Piotrowski September 24, 2013 NLPHist 14/1 OCR for historical documents For historical languages corresponding resources do not exist and are not easy to create (but see the work of the IMPACT project) Spelling variation makes construction of lexical resources difficult Typefaces used in historical prints are often hard to read Abbyy offers support for 19 th century documents in blackletter typefaces (previously in FineReader XIX, now in Recognition Server) No general solution for older texts, though Piotrowski September 24, 2013 NLPHist 15/1

7 OCR quality for historical documents 10 How good does OCR work for historical documents? It depends Character error rate vs word error rate Experiment for Collection of Swiss Law Sources (high image quality, roman typeface): 8665% word accuracy 1097 liath und Gemeinde von Zürich treffen mit Rath und Gemeinde von Winterthur eine Übereinkunft betreffend Zidaming der Angehörigen der einen Stadt vor die Gerichte der andern ah Zeuqen und Abschaffung des Schiddverltafies 1254 Mat 27 Zürich Allen die disen brief ansecbciit oder hörent lesen, künden wir der rat und al di'i menige der 1burger von liath Zürich, und das Gemeinde wir mit von dem Zürich rate und treffen mit der mitmenige Rath der undburger Gemeinde von Wintertur von Winterthur mit uns eine übereinkomen Übereinkunft sin, betreffend das wir die Zidaming gesetzte, der die Angehörigen wir gegen enander einen hatten, Stadt beidenthalb vor die abe- und si Gerichte gelassen haben der andern und haben ah Zeuqen gesetzet und alsus, Abschaffung das die burger des Schiddverltafies von Winter tur und die bi inen wonhaft und gesessen sint in ir stat und in ir getwinge, über unser burger und über die, die bi uns in unser 1254 Mat 27 Zürich»tat und in i'msevm getwinge wonhaft und gesessen sint, an ir geriehten und au ünsern gezügen wol mugen sin, und das hinwider unser burger und die in unser stat und in ünsern getwingen wonhaft und gesessen sint, dasselbe recht haben über ir burger und über die, die in ir stat und in ir getwingen wonhaft und gesessen sint an iren geriehten und an ünsern Och soll man cntwedernthalb Piotrowski niemane heften und September kunbern 24, 2013 umb gelt mit NLPHist vachenne ald mit verbiettenne sin gut, er si dann gelte 16/1 oder bürge, und dennocht sol man denselben nicht heften mit vachenne, won daz man im sin gut wol künhern und verhielten mag mit dem gericht und inrethalb gerichte Wer aber daz von deweder stat ein burger us der andern stat burger umb gelt klagete und er da rechtlos vor dem gerichte Example gelassen wurde, page mag er das from beweren Collection vor siuem gerichte, of das Swiss sol man dem Law andern Sources gerichte künden, und swie daz nicht widertan wirt inret acht tagen, so mag [man]") darumb heften an dem gut des gerichtes burger und gesessen lüte, do er da rechtlos gelassen ist Dise gesetzte sint beschechen durch 752 liebe und durch früntschaft und Nr durch besserunge der geriehten ietwcderhalb Und herüber das 467 dis Ammann war und sij Rat und von stät Davos belib, fordern ihre so Bundesgenossen haben wir von unser insigel au disen brief ze einem offennen urküud h ) Chur auf, an das von den Bünden beschlossene Recht im Streit zwischen brief Maienfeld ward und Peter geben Spiner zc einen Zürich unverdachten Mann do zu man zalt von gottes gebärt zwölf hundert und filnftzig jär Dirre schicken und darnach in dem vierden jare an der mitwachen vor pfingsten, do och dise gesetzte geschneit Juni 24 [Davos] Unsser willig dienste und alles gøtt F±rsichtigen, wissen herren, wir f gen ±ch ze wissen, Copie das die santbotten aus dem des meren Anfang tayls vom goczhus des und 15 ouch Jahrhunderts von gemeynen nach dem damals noch vorhandenen, jetzt verlorenen gerichten uff Tavas ze tagen gewessen synd und da mit gemeynem raut ain Original recht tag geseczt zu ist, Wintevthur; als zwschent in dem sog Quodlibet d h einem Copialbuch der Zürcher Stadtseh reibe rei, welches denen von Meyenveld und Petter Spiner des um 10 andren Mitte tails, des das selb 14 recht Jahrhunderts beseczst ist von den Acht begonnen Gerichten und etlichen und vom rom Stadtschreiber Michael Stehler, genannt Graf, um 1432 goczhus, als dan uff dem tag mit gemeynem ratt beschechen ist Und darumb, lieben herren, fortgesetzt bitten und manent wurde; wir ±ch ayds St und A eren, Z, als hoch B wir III ±ch ze 2 manen fol 183 Von Grafs Hand ist diese Uikünde, sowie unter 1442 andern hand, das auch ir ±ns ain die wissen, vorhergehende, unverdachten man schickent bei zø demselben welcher rechten, ausdrücklich steht «scriptum ext 31ich» (aet) lieber dieser darumb wir versprechen umb sold und zerung als dan gebûrlich jn sœlichem Urkunde steht: *Ahgschrift des briefs, so die dingen ist ze tøn, und das der an dem næchsten sunentag nach sant Peters tagvon Wintertur ze 15 von uns haut», also nach dem Orig in Wintev- thur, nacht so unverzogenlich dass das hie uff Zürcher Tavas an der herbrig Exemplar sy und das nit schon silasse Und damals lie- verloren war, ben herren, tønd hierjn als wir ±ch gancz wol getr±wen, das wellent wir mit gøttem umb ±ch verdienen Geben uff sant Johans tag paptyste, anno domini lxix Amman Spätere und rautt uff Copien: Tavas etc Stadt Archiv Winterthur, Copiebuch III 33 (von a" 1767) mit Bemerkung, dass die Copic 20 Original: «-von (A) ASC, einem Ratsakten, offener Herrn Brief, rückseits und Siegelspuren Fründ Adresse: in Den Zürich f±rsichtigen, commumeieret worden» Stadtbibl Winterthur, Ms 2ß wissen burgermayster und rautt der statt Chur, ±nsren lieben herren und getr±wen puntgenossen etc 2> 334, von Goldschmied 1765 Druck: JM II, Nr 21 Regest: JM I, Nr 34 Druck im Auazwff bei Hotz: histor-jurist Beitrage zur Geschichte der Stadt Winterthur, 1868 p 'J!>, a abev A nur nach der Copie von 1767, obwohl die älteste Copie in dem von Hotz selbst verwalteten Archive lag 1 2 Juli [1469] a) Ist /war gestrichen, doch wohl mit Unrecht, b) Hier fehlt: «gehenkt» oder e a Wort 468 Schreiben der zu Davos versammelten Boten der Elf Gerichte an den Bischof von Chur, an Bürgermeister und Rat der Stadt Chur und das gemeine Gotteshaus über die rechtlichen Bedingungen, unter denen sie an das Gotteshaus überzugehen bereit wären 1470 November 26 [Davos] nser willig undertenig dienste zø voran Hochwirdiger virst und genediger herr, ouch hochgelerter, erwirdiger herr und lieben herren und getr±wen bundgenossen Wir haben ze gøtten tail wol vernomen die maynung und abrednust, so dan usgangen ist von ±weren genaden und wishait gegen ±nsren bottenn, daby wir al- Piotrowski September 24, 2013 NLPHist 17/1

8 Improving OCR quality Difficult to get inside (commercial) OCR systems How to improve OCR results for historical texts? NLP-related ideas ( Holley 2009): Use more than one OCR system and voting to pick the best results ( Handley/Hickey 1991) Use special dictionaries in the OCR process Manual correction of OCR-produced text Language modeling during or after OCR processing Piotrowski September 24, 2013 NLPHist 18/1 Manual text entry If the texts have already been digitized (eg, in a previous project or for Project Gutenberg), error checking and correction may be required to achieve the desired quality If the texts are available in printed form with good print quality and in roman type, optical character recognition (OCR) or double-keying may be options; for both approaches the documents have to be scanned first If the texts are printed, and the print quality is low or the texts are printed in blackletter types, it depends on the language and the desired quality whether OCR (with post-processing) is still an option, otherwise qualified double-keying is likely to be required If the texts are handwritten, double-keying by qualified personnel is currently the only option Piotrowski September 24, 2013 NLPHist 20/1

9 Double-keying Double-keying: 1 Independent keyboarding by (at least) two typists 2 Automatic comparison of resulting texts 3 Error correction Often associated with offshore text entry (outsourcing of keyboarding to service providers in countries with lower wages, in particular to the Philippines, India, or China) In fact: General method for quality control Piotrowski September 24, 2013 NLPHist 21/1 Double-keying (Offshore) double-keying can be a cost-effective and highly accurate solution If OCR is also feasible, weigh researcher time required for post-ocr correction against cost of double-keying Double-keying can achieve > 999% accuracy (99995% accuracy means less than one error per 20,000 characters) Cost depends on complexity, amount of text, and desired quality Some numbers: Text entry 040 to 070 EUR per 1000 characters Quality control 015 EUR per 1000 characters Postprocessing: Data format conversion, semantic markup, etc (as for OCR output) Book page 3440 characters Piotrowski September 24, 2013 NLPHist 22/1

10 Expert transcription F Le Bourgeois, H Empto Diversity of fonts and obsolete characters Material unsuitable for OCR, eg, handwritten documents, blackletter older than 19 th century, anything that requires human judgment expert transcription te transcription of a book can be achieved in 6 h ook of 200 pages with 2,000 characters per page by l labeling of all the patterns from the dictionary figures have been coarsely evaluated by observtranscription of several books by different users d transcription generally increases the transcripeed by 98%, which is the average redundancy rate red on Renaissance books It means that only 2% haracters Transcription must be manually entered tools obtain the iption of the entire book This approach is possiause we have used a precise pattern comparison entire book, which does not tolerate substitution Links between full-text words and their location mage are also stored in the compressed file More- 0% of the pattern dictionary concerns rare charatterns that occur only once in the entire book enerally represent degraded characters or rarely or broken characters If we manually transcribe of the pattern dictionary corresponding to the 000 patterns from the dictionary, which describe st frequent character patterns, we obtain a correct Spain iption of 80% of the entire book in 3 h This rate cient for a query with a search text engine that s missing characters The encoding of obsolete ters from the Renaissance is not solved because is still no standardized Unicode description for haracters Therefore we use substitution letters e these special characters with a loss in readabilthe transcription Moreover, the lack of precise ntation of words influences the transcription readbut not the query by keywords We have observed rs behavior during the transcription by using the A number unexpected applications were found dur ing the DEBORA project For example, we noticed that our compression scheme based on character pattern redundancies is sensitive to imperceptible variation o the typesets used by the typographer The variations in the character pattern redundancy rate provide interest ing information on printing regularity and book manu facturing It describes the successive corrections of the books and provides information to compare differen editions of the same book A future project will use our character pattern comparison algorithm to study the different typesets used in France and authenticate the different publisher and book origins during the Renais sance Tools may help transcribers: Computer-assisted transcription Piotrowski September 24, 2013 NLPHist 23/1 Several projects have produced tools to support transcribers ( CAT, computer-aided transcription), for example: AGORA for the Bibliothèques Virtuelles Humanistes project, DEBORA CAT tool for the DEBORA EU project 5 Versatile format suited for digitized books The DEBORA project is an exploratory project tha has attempted to develop new ideas to improve acces sibility to digitized books In this context, we studied all existing formats that can manage digitized book as heterogeneous data (structures, text, images, links annotation) containing a high density of interconnected information The file format must also allow querying and editing any element of the book content Image formats such as TIFF, XIFF, PNG, and JPEG2K, etc printing formats such as PDF and PS, and edition for mats such as RTF/Word, LaTeX, SGML/TEI, XML are not suitable for simultaneously managing images, text annotations, links, and structures On the other hand XML METS is an appropriate file format that can de scribe documents in both image and text mode as wel GIDOC for idoc (Interactive Analysis, Transcription and Translation of Old Text Documents), Instituto Tecnológico de Informática, OTTO for the Reference Corpus Middle High German ( ) (Ruhr University Bochum, Germany) T-PEN (Saint Louis University, USA) Piotrowski September 24, 2013 NLPHist 24/1

11 DEBORA he RenAissance 217 els gives description For the DEBORA project, we developed a tagged images The parsing file format, similar to XML METS, but encoded in a akes rapid browsing binary file, to reduce storage and increase management ML-METS manages speed [24] Like RTF/Word, our format uses a highly ption files and image compressed standalone file A single file can contain ween image descriptandalone idoc: file format GIDOC tool books Accessing the documents, editing the document s only a few elements, several images of pages, or several is easier to maintain physical layout and logical structure, managing images digitized documents and their descriptions, and editing annotations are all Prototype system for computer-assisted transcription of opment of digital lifields for the DIA gressive transmission and display for remote consulting improved The file format can be decomposed for pro- handwritten historical documents Integrates interactive-predictive Figure 34 gives an example page layout of a book s analysis, description text lineusing detection, and transcription Piotrowski September 24, 2013 NLPHist 25/1 Based on GIMP Open source ( Piotrowski September 24, 2013 NLPHist 26/1

12 idoc: GIDOC tool Piotrowski September 24, 2013 NLPHist 27/1 idoc: GIDOC tool Piotrowski September 24, 2013 NLPHist 28/1

13 OTTO Transcription tool designed for diplomatic transcription of historical text Focus on easy entry and instant rendering Supports management of transcription projects involving distributed, collaborative work Piotrowski September 24, 2013 NLPHist 29/1 OTTO Piotrowski September 24, 2013 NLPHist 30/1 Figure 1: Screenshot of the text editor

14 T-PEN Web-based tool Performs automatic line and column segmentation, then provides transcriber with text entry box synchronized with the lines in the manuscript image Lines can be annotated Integrates project management facilities for collaboratively working on transcriptions Transcriptions can be exported in a variety of formats, including TEI Can be used on the project s server at Saint Louis University Planned to be released as open-source software Piotrowski September 24, 2013 NLPHist 31/1 T-PEN Piotrowski September 24, 2013 NLPHist 32/1