Google’s poor decisions will accelerate the implementation of alternative social sites such as http://bitcoinmegaphone.com/ and http://akasha.world/.
Hunting the hunters
Displaylink setup
Recently I purchased a model# 131-00001-35 Displaylink iClever docking station to use with my laptop.
To properly configure, start by downloading and installing the drivers and xrandr. Check the system configuration:
1 | #pacman -S dkms |
Next start the display link service and list available monitors. setprovideroutputsource activates the second display.
1 |
|
With the display running, xrandr will list available resolutions and refresh rates.
1 |
|
Screen 0 “eDP1” is my laptop screen with a resolution of 1600x99 and a refresh rate of 60 MHz.
Screen 1 “DVI-I-1” is my external display with a resolution of 1280x960.
Resolution can be set/changed with the xrandr command
1 | ;;;external |
Upon reboot, changes are lost. To automatically activate on boot, the systemctl displaylink.service command must be started as a service.
Systemctl requires root privileges.
1 | # systemctl enable displaylink.service |
The xrandr command needs to be executed after the X system is initialized during the boot process. Create an executable file with the required commands:
1 | #!/usr/bin/env sh |
Make the file executable:
1 | $chmod +x ./displaylink.sh |
Go into settings > settings manager > session and startup > application autostart > add
Add an autostart with the following parameters:
Name: Displaylink
Command: bash /home/<myaccount>/displaylink.sh
With this configuration I need to be docked during boot so the second display can be recognized. If I dock after boot, I can manually execute the ./displaylink.sh command to activate the monitor. Without additional configuration I can close the lid of the laptop, inactivating the built in screen, but retaining signal to the external display.
clockwork-orange
Remake the parents into domesticated animals, and the wildness you fear so much reappears in their offspring as if by spontaneous generation.
Sentiment analysis - DT matrix
As I work with various packages related to text manipulation, I am beginning to realize what a mess the R package ecosystem can turn into. A variety of packages written by different contributers with no coordination amongst packages, overlapping functionality, colliding nomenclature. Many functions for “convenience” when base R could do the job. I also noticed this with packages like dplyr. I have commenced learning dplyr on multiple occasions only to find I don’t need it - I can do everything with base R without loading an extra package and learning new terminology. The problem I now encounter is that as these packages gain in popularity, code snippets and examples use them and I need to learn and understand the packages to make sense of the examples.
In my previous post on text manipulation I discussed the process of creating a corpus object. In this post I will investigate what can be done with a document term matrix. Starting with the previous post’s corpus:
1 |
|
There are a variety of methods available to inspect the dtmatrix:
1 | > dtm |
Note the sparsity is 81%. Remove sparse terms and inspect:
1 | > dtms <- removeSparseTerms(dtm, 0.1) # This makes a matrix that is 10% empty space, maximum. |
Now sparsity is down to 4%. Calculate word frequencies and plot as a histogram.
1 | > freq <- sort(colSums(as.matrix(dtm)), decreasing=TRUE) |
We can use hierarchal clustering to group related words. I wouldn’t read much meaning into this for Picnic, but it is comforting to see the xml/html terms clustering together in the third group - a sort of positive control.
1 |
|
We can also use K-means clustering:
1 | library(fpc) |
Back here I didn’t mention that when creating the epub, it would display fine on my computer, but would not display on my Nook. A solution was to pass the file through Calibre. I diff’ed files coming out of Calibre with my originals but was not able to determine the minimum set of changes required for Nook compatibility. You can download the Calibre modified epub here, and the original here. If you determine what those Nook requirements are, please inform me.
Sentiment analysis - Corpus
In a previous post on text manipulation I discussed text mining manipulations that could be performed with a data frame. In this post I will explore what can be done with a corpus. Start by importing the text manipulation package tm. tm has many useful methods for creating a corpus from various sources. My texts are in a directory as xhtml files, one per chapter. I will use VCorpus(DirSource()) to read the files into a corpus data object:
1 | > library(tm) |
The variable “corp” is a 17 member list, each member containing a chapter. tm provides many useful methods for word munging, referred to as “transformations”. Transformations are applied with the tm_map() function. Below I remove white space, remove stop words, stem (i.e. remove common endings like “ing”, “es”, “s”), etc.:
1 | corp <- tm_map(corp, removeWords, stopwords("english")) |
A corpus object allows for the addition of meta data. I will add two events per chapter, which may be useful as overlays during graphing:
1 |
|
The corpus object is a list of lists. The main object has 17 elements, one for each chapter, but each chapter element is also a list. The “content” variable of the list is a list of the original xml file contents, with each element being either xml notation, a blank line, or a paragraph of text. Looking at the second chapter’s contents corp[[2]]$content is a list of 18 elements. The first paragraph of the chapter begins with element 6:
1 | > length(corp[[2]]$content) |
This corpus is the end of the preprocessing stage of the document and will be the input for a document term matrix discussed in the next post
PAHR Sentiment Network
In my previous post on sentiment analysis I used a dataframe to plot the trajectory of sentiment across the novel Picnic at Hanging Rock. In this post I will use the same dataframe of non-unique, non-stop, greater than 3 character words (red line from an earlier post) to create a network of associated words. Words can be grouped by sentence, paragraph, or chapter. I have already removed stop words and punctuation, so I will use my previous grouping of every 15 words in the order they appear in the novel. Looking at my dataframe rows 10 to 20:
1 | > d2[10:20,] |
You can see the column “group” has grouped every 15 words. First I create a table of word cooccurences using the pair_count function, then I use ggraph to create the network graph. The number of cooccurences are reflected in edge opacity and width. At the time of this writing, ggraph was still in beta and had to be downloaded from github and built locally. The igraph package provides the graph_from_data_frame function.
1 |
|
Lets regroup every 25 words:
1 |
|
And now include only words with 5 occurences or more:
1 |
|
PAHR Sentiment Trajectory
In my previous post on sentiment I discussed the process of building data frames of chapter metrics and word lists. I will use the word data frame to monitor sentiment across the book. I am working with non-unique, non-stop, greater than 3 character words (red line from the previous post). Looking at the word list and comparing to text, I can see that the words are in the order that they appear in the novel. I will use the Bing sentiment determinations from the tidytext package to annotate each word as being either of positive or negative sentiment. I will then group by 15 words and calculate the average sentiment.
1 | ##make a dataframe of all chapters |
Plot as a line graph, with odd chapters colored black and even chapters colored grey. I also annotate a few moments of trauma within the narrative.
1 | library(ggplot2) |
We can see that the novel starts with a positive sentiment - “Beautiful day for a picnic…” - which gradually moves into negative territory and remains there for the majority of the book.
Does sentiment analysis really work? Depends on how accurately word sentiment is characterized. Consider the word “drag”:
1 | > d2[d2$word=="drag",] |
There are many instances of the word drag annotated as negative. Consider the sentence “It’s a drag that sentiment analysis isn’t reliable.” That would be drag in a negative context. In Picnic, a drag is a buggy pulled by horses, mentioned many times, imparting lots of undeserved negative sentiment to the novel. Drag in Picnic is neutral and should have been discarded. Inspecting the sentiment annotated word list, many other examples similar to drag could be found, some providing negative, some positive sentiment, on average probably cancelling each other out. Even more abundant are words properly annotated, which, on balance may convey the proper sentiment. I would be skeptical, though, of any sentiment analysis without a properly curated word list.
In the next post I will look at what can be done with a corpus.
