Installation instructions for the various programs I use with EMACS

WindowsXP

R

Download and install R from Cran. I am using version 2.11.1. Sequence manipulation algorithms require the seqinr package v2.0-9 to be installed in R. Launch the gui and select Packages/Install Packages… Select a Cran mirror, then select the package “seqinr”. Packages can also be installed from the command line, once you have R running in EMACS. Use the comand:

1

install.packages("seqinr")

EMACS

Install EMACS. I am using version 23.2

Set up some speed keys:

1
2
3
4
5
6
7
8

(defun load-and-run-fave-file()
  (interactive)
  (find-file "~/owncloud/markov-elear/src/markov_elear/generator.clj"))


(global-set-key (kbd "<f6>") 'load-and-run-fave-file)
(global-set-key (kbd "<f10>") 'primer-db-main)

ESS

Install ESS. I am using version 5.8. Unzip the files as a subdirectory to c:/emacs-23.2/site-lisp. You must include in your .emacs file:

1

(require 'ess-site)

Start EMACS. If ESS and R have been properly installed and configured, you should see:

“Finding all versions of R on your system…”

in the minibuffer during startup. Launch R with the command “M-x R”. You should be able to enter R commands at the > prompt and see the output.

To make working with R more convenient, I include the following in my .emacs file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

;;__________________________________________________________________________
;;;; R  Setup
(require 'ess-site)

(defun my-ess-mode ()
  (interactive)
  (cond ((string= "R" ess-dialect)
	 (message "my-r-mode")
	 ;; customization for ESS[R] here
	 (define-key ess-mode-map '[f5] 'ess-eval-region-and-go))))

;and then hook this into the appropriate place:

(add-hook 'ess-mode-hook 'my-ess-mode)    

;;__________________________________________________________________________
;;;;    Programming - Elisp

(add-hook 'emacs-lisp-mode-hook
	  '(lambda ()
	     (interactive)
	     (require 'eldoc)
	     (turn-on-eldoc-mode)
	     (define-key emacs-lisp-mode-map [f5] 'eval-region)
	     (define-key emacs-lisp-mode-map [f4] 'eval-buffer)))

This allows me to use F5 to evaluate a region whether I am in R or Lisp. I also define the following function key shortcuts to streamline the loading of different files of interest. If you decide not to define these keys, you will need to load the files manually using the command “M-x my-favorite-file”.

1
2
3
4
5
6
7

(global-set-key (kbd "<F6>") 'load-and-run-fave-file)
(global-set-key (kbd "<F10>") 'exe-my-fave-method)

(defun load-and-run-fave-file()
  (interactive)
  (load "my-favorite-file.el"))

If ess is not finding R, check your PATH variable and make sure the R executable is listed.

Clustalw2

Install clustalw2 which performs the sequence alignments. Download the clustalw-2.0.11-win.msi file and allow it to install in c:\Program Files.

Package.el

Try to use Melpa stable only. At least for Clojure compatibility, MELPA unstable introduces bugs.

update MELPA packages (note capital U)

1

M-x package-list-packages U x  

1

M-x package-refresh-contents

Init.el

I switched from ~/.emacs to ~/.emacs.d/init.el
Furthermore I want all of .emacs.d to be in my ~/syncd directory for easy cloud backup.
I need to change the properties of the launch icon to redirect to ~/syncd/.emacs.d/init.el and set the user-emacs-directory variable early in init.el

Properties of the launch icon can only be modified by root so sudo edit /usr/share/applications/GNU Emacs 25 (GUI)

#nautilus
navigate to /usr/share/applications/GNU Emacs 25 (GUI) and chage properties to: /usr/bin/emacs25 %F -q –load “/home/mbc/syncd/.emacs.d/init.el”

%F allows multiple files as arguments.

The first line in init.el is (setq user-emacs-directory “/home/mbc/syncd/.emacs.d/“)

11/01/2019

Working with emacs26

#nautilus
navigate to /usr/share/applications/emacs.desktop and change the line to:
Exec=/usr/bin/emacs %F -q -load /home/mbc/syncd/.emacs.d/init.el

Useful Packages

eldoc: provides method documentation

company: auto completes

ido: provides selection lists e.g. for navigating directories

Principles of Population Genetics, first edition, was the text for Genetics 705 / Plant Science 705 graduate course at State U. I took this course back in 1983 to satisfy requirements for the Master of Science degree in Genetics. Being an early version of the book, there are few of the molecular biology and genomics distractions present in the more recent versions.

I work out many of the problem sets using R.

Recommended texts:

Installation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

>pacman -S guile  ;;installs Guile 2.0.9
>wget https://aur.archlinux.org/packages/gu/guile-ncurses/guile-ncurses.tar.gz
>mv ./guile-ncurses.tar.gz ~/builds
>cd builds
>tar -xzvf ./guile-ncurses.tar.gz
>cd guile-ncurses
>makepkg
>pacman -U guile-ncurses-1.4-1-any.pkg.tar.xz
>pkg-config guile-2.0 --variable=extensiondir
    /usr/lib/guile/2.0/extensions
>mv /usr/local/lib/* /usr/lib/guile/2.0/extensions  ;;move libraries to expected location
>cat ~/.bash_profile
    #
    # ~/.bash_profile
    #

    [[ -f ~/.bashrc ]] && . ~/.bashrc

    export GUILE_LOAD_PATH=${GUILE_LOAD_PATH}:/usr/share/guile-ncurses:/usr/share/guile/site/2.0/ncurses


sudo apt-get install guile-2.0-dev libunistring0 libunistring-dev

Configuration

pkg-config guile-2.0 –variable=extensiondir

Guile-ncurses

download into ./Downloads

1
2
3
4
5
6
7

cd Downloads
tar -xzvf ./guile-ncurses.tar.gz
cd guile-ncurses
./configure --with-guilesitedir=/usr/share/guile/site/2.0

make
make install

init.el

install geiser

;__________________________________________________________________________
;;;; Guile/Lisp Setup
;;(load-file “~/.emacs.d/elpa/geiser-20170325.1956/geiser.el”)

Owncloud will not back up certain files, such as those beginning with underscores. Ftpbox is a hack that allows me to back up all files manually, using batch files. A better option is rsync. Osync for bidirectional sync.

Installation

1
2

#apt-get update
#apt-get install lftp

Batch files

1
2
3
4
5
6
7
8
9
10
11

#!/bin/bash
HOST='mysite.net'
USER='ftpbox@mysite.net'
TARGETFOLDER='/home/mbc/ftpbox'
SOURCEFOLDER='/'
 
lftp -f "
open $USER@$HOST
lcd $SOURCEFOLDER
mirror --verbose --delete --only-newer $SOURCEFOLDER $TARGETFOLDER bye"  

1
2
3
4
5
6
7
8
9
10
11
12

#!/bin/bash
HOST='mysite.net'
USER='ftpbox@mysite.net'
TARGETFOLDER='/'
SOURCEFOLDER='/home/mbc/ftpbox'
 
lftp -f "
open $USER@$HOST
lcd $SOURCEFOLDER
mirror --reverse --delete --verbose --only-newer $SOURCEFOLDER $TARGETFOLDER bye"

put in ~/.profiles for execution on login

for execution at shutdown:

sudo cp ./ftpbox-upload.sh /etc/init.d/
sudo update-rc.d ftpbox-upload.sh start 03 0 6

0 is for shutdown
6 is for reboot
03 puts it in the middle of the scripts run
update-rc.d creates a symlink

Run on login, logout, cron

~/.bash_profile or ~/.bash_login do not exist on Ubuntu

Use ~/.profile for login scripts and append:

~/ftpbox-download.sh

Use ~/.bash_logout for logout scripts

~/ftpbox-upload.sh

crontab -e to edit crontab
add 1 line per command

*/10 * * * * /home/dad/ftpbox-upload.sh

chmod 600 ~/.netrc ;read and write by user only

**~/.netrc **
machine www..net login ftpbox@mysite.net password

Software installation

echo ‘deb http://download.opensuse.org/repositories/isv:/ownCloud:/desktop/Debian_7.0/ /‘ >> /etc/apt/sources.list.d/owncloud-client.list

apt-get update
apt-get install owncloud-client

Local folder: /home/mbc/owncloud
must modify network folder to use ~/owncloud/documents
When asked for network folder type “/“

server: https://www.mysite.net/owncloud

Server configuration

Database: mbc_owncloud
user: mbc_owncloud
password: mypassword

owncloud server user: mbc
password: mypassword

database host:localhost

create an admin user: mbc

I am only using the MySQL method
copy setup-owncloud.php to /public_html then go to

www.mysite.net/setup-owncloud.php

To use WebDav with Zotero

www.mysite.net/owncloud/remote.php/webdav ( /zotero is automatically appended)
you will be prompted to create directory

Client configuration

Zotero

do not export, backup the database directory to retain metadata

~/.zotero/zotero/3xdb…./zotero…
the “pipes” subdir will not copy, be sure to compensate

www.mysite.net/owncloud/remote.php/webdav

The QPix software identifies a single region per plate (two regions total) and randomly picks a user defined number of colonies from each region. Ideally one would want to define 96 regions per plate so that the number of colonies per region can be specified - not allowed. However each colony’s location (X|Y deck coordinate) is recorded in an XML log file.

With a bit of hacking, the well location can be assigned to each colony. In this post I will process the XML log file to extract X|Y coordinates for each picked colony, associate those coordinates with wells, associate clone IDs with wells and ultimately associate the clone ID with a picked colony.

See this post for details concerning the input files and custom methods.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

rm(list=ls(all=TRUE))
library(XML)

setwd('~./qpix2')
#this is the files that indicates the clone IDs of the VH, VL fragments that
#have been cloned - the ID of the transformation mix
assoc.file <- "amplicon-plate-map.txt"
assoc <- read.table( paste(getwd(),"input", assoc.file, sep="/"), skip=0, header=TRUE)

 > assoc
         id dest.plate dest.well
1  PL101-VH  amplicons       A01
2  PL102-VH  amplicons       B01
3  PL103-VH  amplicons       C01
4   PL18-VH  amplicons       D01
5   PL28-VH  amplicons       E01
6   PL38-VH  amplicons       F01


##extract the runid as six integers
qpix.filename <-  "QSoft Log 2014-02-11 105201.XML"
runid <- substring( qpix.filename, 22, 27) 
> runid
[1] "105201"

Supply plate names of the plates in which the picked colonies were deposited. This is not needed if the plate names are typed directly into the software. For this example I will assume they are not. Enter as many destination plates as exist, the script will adjust

1
2
3
4

dest.plate <- list()
dest.plate[[1]] <- "PL140211c"
dest.plate[[2]] <- "PL140211d"

Use region 2 well H01 as the “origin”. Enter calibration block values for well G01. From G01 calculate H01 using hard coded offsets.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

g01.x <- 114278
g01.y <- 241732
xstart <- g01.x + 2580
ystart <- g01.y - 8781
> xstart
[1] 116858
> ystart
[1] 232951

d <- xmlTreeParse( paste(getwd(), "/qpix-logs/", qpix.filename, sep="") )
top <- xmlRoot(d)
##top[[11]] has the "contents"; should be as many deposits as destination plates
>names(top[[11]]) 
     routine      imaging   colonydata      deposit      deposit 
   "routine"    "imaging" "colonydata"    "deposit"    "deposit" 
> 

Retrieve XML log files off the Qpix post run. Colony data is in //contents/colonydata. The XML can be easily visulaized by opening with a browser. Colonydata contains X|Y location as well as region. Read it in and create a dataframe.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

colony.data <- getNodeSet(top, "//contents/colonydata")
d2 <- data.frame(t(sapply( xmlApply(colony.data[[1]],  xmlAttrs),c)), stringsAsFactors=FALSE)
# region-image-colonyid uniquely identifies a colony
# not sure why image is needed
d2 <- d2[,c("colonyid","image","posx","posy","region")]

 > head(d2)
  colonyid image   posx   posy region
1        0     5 159286 385817      1
2        1     5 159569 403677      1
3        2     5 160654 385588      1
4        3     5 160761 388800      1
5        4     5 160854 371823      1
6        5     5 161284 390663      1
> 

Calculate centers of each column along X axis.
Offset between regions 1 and 2 is 108174um.
The custom method getDeckWell defined here
Add the source well to the dataframe.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

x.cent <- seq( xstart, by=9100, length.out=12)
yr1.cent <- seq( ystart+108174, by=9100, length.out=8)
yr2.cent <- seq( ystart, by=9100, length.out=8)

src.well <- rep( NA, nrow(d2))

for( i in 1:nrow(d2)){
  src.well[i] <- getDeckWell( as.numeric(d2[ i, "posx"][[1]]), as.numeric(d2[ i, "posy"][[1]]), as.numeric(d2[ i, "region"][[1]])) 
}
d2 <- cbind( d2, src.well)

> head(d2)
  colonyid image   posx   posy region src.well
1        0     5 159286 385817      1      C06
2        1     5 159569 403677      1      A06
3        2     5 160654 385588      1      C06
4        3     5 160761 388800      1      C06
5        4     5 160854 371823      1      E06
6        5     5 161284 390663      1      C06
> 

d2.wid <- merge( d2, assoc, by.x="src.well", by.y="dest.well")

> d2.wid
          id colonyid src.well
1   PL101-VH      153      A01
2   PL101-VH      162      A01
3   PL101-VH      157      A01
4   PL101-VH      166      A01
5   PL101-VH      160      A01
6    PL44-VH      208      A02
7    PL44-VH      181      A02
8    PL44-VH      206      A02
9    PL44-VH      187      A02
10   PL44-VH      171      A02

Now we have associated an id based on well location to each colony.
src.well here is the well the clone came from in the transformation plate.
All we need here is colonyid assigned by Qpix and my clone id
src.well is the transformation well, keep that so you can
retrieve colonies that weren’t recovered by the Qpix.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

d2.wid <- d2.wid[,c("id","colonyid","src.well")]
deposit <-getNodeSet(top, "//contents/deposit")

deposit.list <- list()
for( i in 1: length(deposit)){
  deposit.list[[i]] <- data.frame(t(sapply( xmlApply(deposit[[i]],  xmlAttrs),c)), stringsAsFactors=FALSE)
}

#may not need to do this if destinations entered in software
for(i in 1:length(deposit.list)){
  deposit.list[[i]]$destination <- dest.plate[[i]]
}

d3 <- do.call("rbind", deposit.list)


d3$location <- sapply(  d3$location, insert0inWell)

> head(d3)
  destination location colonyid              source region
1   PL140211c      A01        0 UIDUO0211-3912159-X      1
2   PL140211c      B01        1 UIDUO0211-3912159-X      1
3   PL140211c      C01        2 UIDUO0211-3912159-X      1
4   PL140211c      D01        3 UIDUO0211-3912159-X      1
5   PL140211c      E01        4 UIDUO0211-3912159-X      1
6   PL140211c      F01        5 UIDUO0211-3912159-X      1
 
d4 <- merge(d3, d2.wid, by.x="colonyid", by.y="colonyid")
d4 <- d4[,c("destination","location","id","src.well")] #src.well is the transformation well 
d4 <- d4[ order( d4$destination, d4$location),]
names(d4)[1:2] <- c("plate","well")
> head(d4)
        plate well      id src.well
1   PL140211c  A01 PL78-VL      C06
137 PL140211c  A02 PL44-VL      A06
51  PL140211c  A03 PL86-VL      C07
86  PL140211c  A04 PL86-VL      C07
95  PL140211c  A05 PL95-VL      D08
104 PL140211c  A06 PL94-VL      C08

results <- as.data.frame(table(d4$src.well))
results <- results[ order( results$Freq),]

> results
   Var1 Freq
1   A01    0
15  C02    0
22  C09    0
26  D04    0
31  D09    0
40  E09    0
41  E10    0

Results shows the count of colonies obtained per clone ID. Some are not present in the collection, as can be seen above where Freq==0. This provides a list of wells that can be plated manually if these are considered valuable colonies.

Pick out 4 candidates (if available) for each clone ID.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

d5 <- do.call("rbind", lapply( split( d4, d4$bdn ), function(x) x[1:4,]) )
d5 <- d5[ !is.na(d5$bdn),]
d5 <- d5[ order( d5$plate, d5$well ),]

##evaluate
set1 <- read.table( paste(getwd(), "103410-picked-plates-map.txt", sep=""), skip=0, header=TRUE)
set2 <- read.table( paste(getwd(), "105201-picked-plates-map.txt", sep=""), skip=0, header=TRUE)

composite <- rbind(set1, set2)
out.file <-paste( working.dir,  "composite.txt", sep="")
write.table( composite, file = out.file, append = FALSE, quote = FALSE, sep = "\t", eol = "\n", na = "NA", dec = ".", row.names = FALSE, col.names = TRUE, qmethod = c("escape", "double"))



results <- as.data.frame(table(composite$src.well))
results <- results[ order( results$Freq),]

plot(results)

set1 and set2 provide the plate maps for the picked plates.
results show the number of colonies picked for each candidate id. For those with 0-2 colonies, manually pick supplemental colonies if available.

results can be plotted to get a sense for the distribution of colony counts for the various candidate clones.

In this post I will provide some setup information for a later post that will discuss extracting clone IDs from a Qpix XML log file. This post consists of two sections, input files and custom methods.

Input files

Plate map file

The amplicon plate map provides the clone ID associated with the PCR product in each well of the 96 well pcr plate. This is the ID that ultimately you would like to associate with a colony.

1
2
3
4
5
6
7
8
9
10
11

assoc.file <- "amplicon-plate-map.txt"
assoc <- read.table( paste(getwd(),"input", assoc.file, sep="\\"), skip=0, header=TRUE)
> assoc
         id dest.plate dest.well
1  PL101-VH  amplicons       A01
2  PL102-VH  amplicons       B01
3  PL103-VH  amplicons       C01
4   PL18-VH  amplicons       D01
5   PL28-VH  amplicons       E01
6   PL38-VH  amplicons       F01

QPix2 XML files

These are the log files retrieved from the Qpix post run.

[Example 1](QSoft Log 2014-02-11 103410.XML)
[Example 2](QSoft Log 2014-02-11 105201.XML)

Useful Methods

getDeckWell()

1
2
3
4
5
6
7
8
9
10
11
12
13
14

getDeckWell <- function( x, y, region){
  #region is 1 or 2
  cols <- c("01","02","03","04","05","06","07","08","09","10","11","12")
  rows <- c("H","G","F","E","D","C","B","A") 
  ##note that if a colony has equal but opposite sign offsets with two columns, this algorithm will fail!!  
  x.offset <- x.cent-x  
  y.offset <- switch( region,
                     "1" = yr1.cent-y,
                   "2" = yr2.cent-y)
  
  paste(rows[ which(abs(y.offset)==min(abs(y.offset)))],
        cols[ which(abs(x.offset)==min(abs(x.offset)))], sep="")

}

getWell96()

The function returns the opposite of what was supplied as a well ID i.e. provide well number get back the three character well ID. Provide three character well ID get back well number.

First create a table associating the three character well ID with well number.

1
2
3
4
5
6
7
8
9
10

> well.ids
   name  n
1   A01  1
2   B01  2
3   C01  3
4   D01  4
5   E01  5
6   F01  6
7   G01  7
8   H01  8

Once the table is defined, lookup either integer or three character well ID using the function argument.

1
2
3
4
5
6
7
8
9
10
11
12

getWell96 <- function( id ){
  n <- 1:96
  row <- rep(LETTERS[1:8],12)
  col <- sort(rep(1:12, 8))
  zero <- c(rep(0,72), rep("",24))
  name <- I(paste(row,zero,col, sep=""))  #I() is as.is to prevent factorization
  well.ids <- data.frame(name, n)

  if( id %in% name) {well.ids[well.ids$name==id,"n"]
                   }else{
                     well.ids[well.ids$n==id,"name"]}
}

insert0inWell()

Work only with three character well IDs e.g. A01, not A1. insert0inWell will insert a “0” in a 2 character well ID.

1
2
3
4

insert0inWell <- function( id){
  if( nchar(id) == 2  ){  paste( substr(id, 1,1), "0", substr(id, 2,2), sep="")  
                        }else{ id }
}

In a previous post I showed how to use Omni plates to titer bacteria. Another useful application of OMNI plates is to plate out transformation mixes. In a high-throughput (96 well) cloning process it is easy to run In-Fusion reactions and transform bacteria in 96 well format. Vendors provide competent cells for chemical transformation in 96 well format. The bottleneck is plating and picking colonies. OMNI plates can be used to plate out 96 individual cloning reactions per OMNI plate.

The bottleneck then shifts to the next activity - picking colonies for sequencing. A Qpix2 colony picker can be used for picking. The Qpix deck will hold 2 OMNI plates using an Omni tray holder:

The software identifies a single region per plate (two regions total) and randomly picks a user defined number of colonies from each region. Ideally one would want to define 96 regions per plate so that the number of colonies per region can be specified - this is not allowed. However, each colonies location (X|Y deck coordinate) is recorded in an XML log file, and with a bit of hacking, the well location can be assigned to each colony.

The first step is to determine the how to interpret the coordinate system of the deck. I start by placing an agar filled OMNI plate on the deck of a BiomekNX liquid handling robot. Load the head with P20 tips and impale the surface of the agar.

The indentations on the surface of the agar are (mis)interpreted by the software as a colony and the X/Y cordinate will be reported. From this data one can determine the X|Y layout of the deck.

Now I can determine the coordinate boundaries for each of the 96 regions of interest:

More generally I can determine the coordinate layour of the entire deck. The coordinate system appears to be upside down, not surprising given that well A01 for each plate is bottom right corner. Apparently the engineers were standing at the back of the instrument during the design process. This has not been corrected in the newer version.

Now knowing the coordinates of the center of each well, I can assign an area to each well, and a well id to each colony that falls within that area.

I will assign an anchor coordinate and calculate everything as an offset to that. What if a PM or service call changes the deck slighly? I would like a way to calibrate. I find that I can use an ABI PCR plate holder as a calibration tool. Place the holder in the OMNI plate, scan, and the image can be used to assign an anchor coordinate. Because row H is too close to the edge of the image, “colonies” (actually the holes in the ABI plate) are not identified, so I use well G01 as my calibrator.

I can then calculate the offset between the ABI plate and the impaled OMNI plate. Should the coordinates of the ABI plate well G01 ever change, the change can be incorporated as an offset in the scripts.

Using this method I can assign source wells to the colonies I have picked and look at the pattern of deposit into the destination plate.

In this image the X|Y coordinates correspond to the OMNI plate e.g. OMNI plate well A01 is in the upper left corner, and the well ID is from the source plate. So the transformation mix from well D12 was deposited at position A01 on the OMNI plate. Looking at the pattern, I see no logic to it. Molecular Devices claims the picking order is optimized for speed.

Here is an image from an advertisement from the early to mid oughts for the object database ObjectStore:

We (colleagues and I) speculated as to when we would be able to query images based on content. Who are these guys? What are they celebrating? What year is it? Where does the celebration take place? TinEye to the rescue:

Four international cocktail experts tasting some new drinks at the Wines, Spirits and Catering Trades Exhibition at Dorland Hall, London. October 30, 1933