Say you have a column which contains categorical variables / factors. Then How can we quickly and confidently identify the differences between different categories?

A very commonly used dataset ‘mpg’ from the package ‘ggplot2’ contains some categorical variables that could easily get started. There is a column in mpg called ‘cty’ which is the miles per gallon for a car while driving in city. And also another column ‘manufacturer’ which contain categorical variable the different manufacturers.

library(ggplot2)
data(mpg)
bymedian = with(mpg, reorder(manufacturer, -cty, median))
boxplot(cty ~ bymedian, data = mpg, varwidth = TRUE)

result = aov(formula = mpg$cty ~ as.factor(mpg$manufacturer))
TukeyHSD(result)

Background Story:

Sir Francis Galton, who is the cousin of Charles Darwin was an English Victoria polymath, eugenicist and statistician. He proved the linear combination of normal distribution is still normal distribution and he is also,the inventor of the triangle device – quincunx. Here, we will talk about the research that he has implemented long time ago(1885).

library(UsingR)
data(galton)
par(mfrow=c(1,1))
plot(galton$parent, galton$child, pch=19, col=”blue”)
m <- lm(galton$child ~ galton$parent)
lines(galton$parent, m$fitted, col=”red”, lwd=3)

First, and the most horrible one when you have so many points:

data = data.frame(x=rnorm(1e4), y=rnorm(1e4))

plot(data, pch=19)

smoothScatter(data)

library(hexbin)

plot(hexbin(data))

library(ggplot2)
ggplot(data, aes(x=x,y=y)) + geom_point(alpha=I(1/3))

1.  zoo: S3 Infrastructure for Regular and Irregular Time Series

2.  Reshaping Data with the reshape Package

3.  Tidy data (submitted only from Hadley Wickham)

Say, you have a time series object that you want to work with. You might want to check if the time series happens to be generated by white noise only, this might happen not only during the interview but also in real life sometime.

If you have three contiguous time points, assuming they are of different values. Then there are 6 permutations to order them.  And four of them form a turning point (PEAK/PIT).

The expectation of number of turning points for a time series generated by white noise is then 2/3 * n, and actually they falls into a normal distribution where the variance is 8/45*n.

So your 5% confidence test should check if the number of turning points is within the range of: 2/3*n +- 1.96*sqrt(8/45*n):

require(pastecs)
data <- rnorm(10000, 0, 1)
plot(data)
limit <- function(n){
print(2/3 * n)
print(2/3 * n – 1.96 * sqrt(8/45 * n))
print(2/3 * n + 1.96 * sqrt(8/45 * n))
}
turnpoints(data)
limit(10000)

OUTPUT

> turnpoints(data)

Turning points for: data nbr observations : 10000 nbr ex-aequos : 0 nbr

turning points: 6657 (first point is a peak) E(p) = 6665.333 Var(p) = 1777.456 (theoretical)

> limit(10000) [1] 6666.667 [1] 6584.026 [1] 6749.308

You can clearly see that the fake time series does actually have 6657 turning points and it has been tested falls into our “BLACK LIST INTERVAL”!

I happen to find a list of all the R base functions, then I cannot help going through some of them. To satisfy my pride, I decided to look in the back order.

1. yinch – xinch

(Move the point in the unit of inch while plotting)

2. edit/vi/emacs/xemacs/xedit

testData <- edit(data)

3. getwd()/write()/unlink()

get the current working directory, write matrix to the working directory, delete the file

4. file()/file_test()/file.access()/file.remove()/file.copy()/file.exist()/file.info()/file.append()/file.symlink()/file.link()/file.path()/file.show()

… all the low level interface to the computer’s file system