In my first post on creating an ebook I discussed the physical manipulation required to convert a paperback book into images and ultimately text files. Now I want to convert the text files into an ebook. Here is the sequence of events:
library(tidytext) library(dplyr) library(stringr) library(readr) library(tokenizers) > d file.name bname chpt eo page img.num pnumber chpteo 1 ch1o-0001.txt ch1o-00011 o 11NA1o 2 ch1e-0010.txt ch1e-00101 e 210NA1e 3 ch1o-0002.txt ch1o-00021 o 32NA1o 4 ch1e-0009.txt ch1e-00091 e 49NA1e 5 ch1o-0003.txt ch1o-00031 o 53NA1o 6 ch1e-0008.txt ch1e-00081 e 68NA1e 7 ch1o-0004.txt ch1o-00041 o 7471o 8 ch1e-0007.txt ch1e-00071 e 8781e 9 ch1o-0005.txt ch1o-00051 o 9591o 10 ch1e-0006.txt ch1e-00061 e 106NA1e
By counting the number of rows associated with each chapter in the dataframe, determine the number of pages per chapter then combine those pages into a list by chapters, 17 chapters total for Picnic. I will not annotate individual pages with page numbers, but will combine all pages into a chapter and let the epub format handle the flow.
# how many pages per chapter chap.lengths <- vector('integer',length=16) for(i in1:16){ chap.lengths[i]<- nrow(d[d$chpt== i,]) }
chapter <- vector(mode ="list",length=16)#there are 16 chapters page.counter <- 1
for(i in1:length(chap.lengths)){#i is the chapter number for(j in1:chap.lengths[i]){#j is the page number per chapter but I need a page counter to span the book chapter[[i]]<-c(chapter[[i]], pages[[page.counter]]) page.counter <- page.counter +1 } }
Remove ligatures, correct misspellings
OCR wil have introduced many misspellings, some of which can be corrected in bulk. I also want to remove ligatures, as this will interfere with word recognition when I am performing spell checking. Finally, the type setting process introduces many hyphenated words at the end of a line of text, to preserve readability. I want to remove these and let the epub flow text instead.
I create the method replaceforeignchars which will replace ligatures and common misspellings. The replacements to be executed are tabulated in the table “fromto”:
chapters2 <- vector(mode ="list",length=16)#there are 16 chapters concat.flag <-FALSE#if True must concatenate last word on sentence to first on next sentence and remove dash
In practice this didn’t work so well. Tokenizing a sentence removes capitalization which then has to be manually corrected. There were also occasions where a line was duplicated and this had to be manully corrected. I decided to remove hyphens manually while editing the text.
Next I print out each chapter as a page with xhtml annotation:
The package qdap provides an interactive method, check_spelling_interactive, for spell checking. A dialog bog will pop up for each unrecognized word in turn, providing you with a pick list of potential corrections or the opportunity to type in a correction manually.
I found that the pick list often did not provide the appropriate choice, capitalization is not preserved, and Picnic has many slang words that forced interaction with qdap too frequently. I decided to read through the text and correct manually.
Here is qdap flagging the French ‘alors’. There are settings for qdap that may improve the word choices available, but I did not spend the time investigating.
Once the chapters have been edited and proof read, it is time to create the epub. An epub is a zip file with the extension “.epub”. It also has a well defined directory layout and required files that define chapters, images, flow control, etc. ePub specifications and tutorials are readily available on line. Here I will show examples of some of the epub contents.
<head><metacharset="UTF-8" /></head> <body><p>Chapter 2</p><p></p> <p> Manmade improvements on Nature at the Picnic Grounds consisted of several circles of flat stones to serve as fireplaces and a wooden privy in the shape of a Japanese pagoda. The creek at the close of summer ran sluggishly through long dry grass, now and then almost disappearing to re-appear as a shallow pool. Lunch had been set out on large white tablecloths close by, shaded from the heat of the sun by two or three spreading gums. In addition to the chicken pie, angel cake, jellies and the tepid bananas inseparable from an Australian picnic, Cook had provided a handsome iced cake in the shape of a heart, for which Tom had obligingly cut a mould from a piece of tin. Mr Hussey had boiled up two immense billycans of tea on a fire of bark and leaves and was now enjoying a pipe in the shadow of the drag where he could keep a watchful eye on his horses tethered in the shade. </p> . . . <p> The four girls were already out of sight when Mike came out of the first belt of trees. He looked up at the vertical face of the Rock and wondered how far they would go before turning back. The Hanging Rock, according to Albert, was a tough proposition even for experienced climbers. If Albert was right and they were only schoolgirls about the same age as his sisters in England, how was it they were allowed to set out alone, at the end of a summer afternoon? He reminded himself that he was in Australia now: Australia, where anything might happen. In England everything had, been done before: quite often by one’s own ancestors, over and over again. He sat down on a fallen log, heard Albert calling him through the trees, and knew that this was the country where he, Michael Fitzhubert, was going to live. What was her name, the tall pale girl with straight yellow hair, who had gone skimming over the water like one of the white swans on his Uncle’s lake? </p><p></p></body> </html>
Once the files are in order they are zipped into an epub. Navigate to the directory containing your files and:
1
zip -Xr9D Picnic_at_Hanging_Rock.epub mimetype * -x .DS_Store
Some of the switches I am using:
X: Exclude extra file attributes (permissions, ownership, anything that adds extra bytes)
-r: Recurse into directories
-9: Slowest but most optimized compression
-D: Do not created directory entries in the zip archive
-x .DS_Store: Don’t include Mac OS X’s hidden file of snapshots etc.
The next post in this series discusses sentiment analysis.
