Friday, January 24, 2014

Add a table of content to HTML files generated from R Markdown

Update November 2014

With the new version of knitr and Rmarkdown, the custom function is not necessary anymore. One can add a yaml at the beginning of a Rmd file:
---
title: "Development scrap concerning the input data"
output:
  html_document:
    toc: true
---



Old content from January 2014

Knitr creator Yihui explained in a comment on this forum how to add a table of content to a Rmd file using the knit2html() function:
library(knitr)
knit2html('docs/clean.example.Rmd', options = c('toc', markdown::markdownHTMLOptions(TRUE)))
I followed the RSTUDIO advice on how to customize markdown rendering.
A .Rprofile  at the root of my project directory with the following content does the tric:
options(rstudio.markdownToHTML =
  function(inputFile, outputFile) {     
    require(markdown)
    htmlOptions <- defaults="TRUE)<br" markdownhtmloptions="">    htmlOptions <- br="" c="" htmloptions="" toc="">    markdownToHTML(inputFile, outputFile, options = htmlOptions)
  }
)
I works, I can now use the RStudio button "knit html" or the shortcut CTRL+SHIFT+H and get an html file that includes a table of content!

Thursday, January 23, 2014

GFPM

Problem: In Windows 7, I was getting UAC message for each .exe component of the GFPM model.
Fix: stop annoying UAC prompts, recommends to use the task scheduler and set highest priviledge for the task. I used it to run a GFPM task. Then created a shortcut to:
C:\Windows\System32\schtasks.exe /RUN /TN GFPM\GFPM
I called this Shortcut "Run GFPM without UAC warning". It seems to work, there were no UAC warnings anymore!

A next step would be to run GFPM under wine in linux. That would make it easier to run more than one simulation. From wine it's not going to be possible to use Excel, so I might have to start the batchfiles after the call to world.xls.

Tuesday, January 21, 2014

R commands

A list of commonly used R commands.

Remove all objects from the workspace:
rm(list=ls())

Yihui Xie wrote that "setwd() is bad, dirty, ugly." Use relative paths instead.

 

Testthat library

Run all tests in a directory:
test_dir("tests")

Wednesday, January 15, 2014

Install R Packages

I recently participated in a training on the use of R to extract data from permanent forestry plots. These is how to install the R packages used in that training:
install.packages(c("doBy", "reshape2", "ggplot2", "GISTools", "lattice", "gstat", "knitr", "raster", "xtable", "rgdal"))
Somehow it didn't install all dependencies for ggplot2, I needed to run:
install.packages('ggplot2', dep = TRUE)

Monday, January 13, 2014

Msysgit makes GNU Bash available on Windows

Msysgit makes Bash available on windows. It is related to MinGW - Minimalist GNU for Windows, itself a fork of cygwin. The version I use on my system is msysgit. With this tool, some linux like command line operations can be run on windows.

 MSYS Git FAQ:
"MSys is an environment for Windows offering a Unix-type shell and a Perl interpreter. Because many parts of Git are still not builtins programmed in C, but instead shell and Perl scripts, Git for Windows needs such an environment. Therefore we ship Git for Windows with a very minimal version of MSys."
Example of commands:
Compute md5sum
md5sum filename
See also:

Thursday, January 02, 2014

Ipython notebook

Start server available on local network:
ipython notebook --ip=192.168.xxx.xxx

GNU-Linux bash shell commands


Linux is the kernel of the operating system on top of which other programs are built. A detailed list of GNU core utilities is available under the command :
info coreutils

 Files 

determine file type and encoding
file filename
list a directory
ls
ls -R #list subdirectories recursively
ls -lh #sizes in human readable format
Find files in subdirectories of the current directory (Quotes are requited to prevent shell command expansion).
find . -name "*.pdf"
find . -mtime 0 # modified in the last 24 hours
Find files in the whole system
locate filename

File and folder compression

Decompress a file
 gunzip file.gz
How do I compress a whole directory?
tar -zcvf archive-name.tar.gz directory-name
Where
  • -z: Compress archive using gzip program
  • -c: Create archive
  • -v: Verbose i.e display progress while creating archive
  • -f: Archive File name
To extract content from the archive in the current directory
tar -zxvf archive-name.tar.gz

Rename files

For example to rename all upper-case .JPG extension into lower-case .jpg extension.
rename 's/\.JPG$/\.jpg/' *.JPG
Change file permission:
chmod a=rwx filename
chmod 777 filename 
Change file permissions recursively:
chmod 755 directoryname
Chmod instructions can be given with characters or numbers, chmod 777 or chmod a=rwx is a question of preference.
  • Some prefer 755 over 777 because giving write access to group and other users could be a security risk. 755 leaves read and execute rights to groups and other users. 755 is visible as "rwxr-xr-x" in ls -l. 
  • The default for document files on Debian seems to be chmod 644, visible as "-rw-r--r--" in ls -l.

Text files

Count the number of lines in a file
wc -l filename.txt
Count occurrences of a word in a file
grep -roh word filename.txt  | wc -w
Remove duplicated lines from a file
awk '!a[$0]++' input.txt
Search with Grep
 grep "text" file.txt
Awk tutorial, for example  filter a large file for lines that have a third field (product code) starting with 44, keep the header line:
awk -F, '$3 ~ /^44/||NR==1' nc201501.dat|less
Regexp match begining of and end of line with ^ and $.

Follow the end of a log file as it is written 
tail -f
See tab and end of line characters in a text file
cat -te filename |less

Manipulate strings in files

Replace strings
first="I love Suzy and Mary"
second="Sara"
first=${first/Suzy/$second}
Replace strings with sed

sed -i  's/pattern/replacement/g' bli.txt
sed -i  's/^.*\://g' input_file.txt # edit file in place
grep EMAIL input_file.txt |sed  's/^.*\://g' > output_file.txt
Replace strings with perl in a git repository

git grep -lz 'readcsvfromgauss'| xargs -0 perl -i'' -pE "s/readcsvfromgauss/readcsvfromgauss0/g"

PDF files

Commands based on the poppler library for PDF manipulation.
Search a text pattern in all PDF files present in a directory:
pdfgrep pattern *.pdf
 Merge multiple PDF into one:
pdfunite in-1.pdf in-2.pdf in-n.pdf out.pdf
Alternatively, pdftk can be used to merge PDF files
pdftk input1.pdf input2.pdf cat output output.pdf

Videos and audio

Install youtube-dl using pip:
 sudo pip install --upgrade youtube_dl
Download a video from youtube :
youtube-dl video_url
Download only the audio in an .mp3 format
youtube-dl --extract-audio --audio-format mp3 video_url

Users

Check your user id
id 
What group do you belong to as a user
groups
Add a user to the super users
adduser username sudo
That user needs to re-log into the shell for the change to take effect.
Add a new user
useradd username
Set a password for the new user
passwd username
Delete a user
userdel username
Show all users
getent passwd
Show all groups
getent group

System

OS release
less /etc/os-release
Disk usage
du -h
Display available space on drives
df -h
Display available RAM memory
less /proc/meminfo

Install a program
sudo apt-get install
 System name
uname -a
file /sbin/init
hostname -f
Start and quit a super user session
su
exit
Last time the system was started
last reboot 
last
Show environment variables
printenv

Job handling

List
jobs
Bring a job to the foreground
fg job_number
Run a job in the background. A command followed by an & will run in the background.

Stop a job
CTRL ^ Z
Quit a job
 CTRL ^ C
Kill a malfunctionning program:
kill process_id
Find a program id with:
ps aux
Kill a graphical program, by clicking on it:
 xkill

Users

Create a new user
adduser user_name
Temporary log in as that user
su user_name
Delete a user
userdel user_name

Secure shell

log into a remote machine
ssh user@remote_machine
Copy a local file to a file on the remote machine
scp local_file_name user@remote_machine:path_to_file/file_name
Copy a file from the remote machine to a local file
scp user@remote_machine:path_to_file/file_name  local_file_name
Copy a full directory (dmouraty) from the remote machine:
 scp -rp user@dest:/path destdirectory

Alias

alias ll="ls -lh"

Based on how can i sort du-h output by size
alias du='du -hd1 | sort -h -r'

You can place those commands in your ~/.bashrc to create a permanent alias.
bashrc:
"You may want to put all your additions into a separate file like ~/.bash_aliases, instead of adding them here directly."

.bash_profile and .bashrc

These are places where a user can turn of the system BEEP :
setterm -blength 0
.bash_profile is executed on login shell, when you login in another tty or when you access a system through ssh. .bashrc is executed on non-login shells when you open a terminal window in Gnome.

Debian Dotfiles
"Now, since bash is being invoked as a login shell (with name "-bash", a special ancient hack), it reads /etc/profile first. Then it looks in your home directory for .bash_profile, and if it finds it, it reads that."
[...] "You may have noted that .bashrc is not being read in this situation. You should therefore always have command source ~/.bashrc at the end of your .bash_profile in order to force it to be read by a login shell.  "
In .bashrc a user can set environment variables, define alias (see above).

Keyboard

bash french blogger recommended a simple shell command to change keyboard layout :
sudo loadkeys fr
fr-keyboard on Debian wiki for a more permanent system configuration and use in GUI apps. Switching between keyboads can then be done with:
setxkbmap de
setxkbmap fr

Information about the system

  • cat /proc/meminfo
  • cat /proc/cpuinfo
  • cat /etc/debian_version
  • lsb_release -a

Shortcuts

Keyboard shortcuts for bash  for example Ctrl+A to go to the beginning of a line.

Documentation