select
mpn,
REGEXP_REPLACE(upper(mpn), ‘[^a-zA-Z0-9]+’, ”) as mpn_norm
from
…

1. m1 <- 1:(10^9)
2. m2 <- 1:(10^9) it exceeded the 12GB physical memory, then starting swapping, I terminated R process.
3. restart R
4. m1 <- 1:(10^8)
5. m2 <- 1:(10^8)
6. m3 <- 1:(10^9), then I did `remove(list=ls())` and it removed the object from the environment but the memory did not get released.
7. gc() clear memory using garbage collection in R

I was following the “High Performance Computing using Rcpp” in Hadley Wichham’s Advanced R. I did an experiment in R which there are two functions, one function myRowSum written in C++, and the other one written in plain R. As you can see from the code, they are very similar, same variable name, same for loop, same logic…etc. However, I have been totally blown away by the difference between total time.

I created a dummy matrix with 100,000 rows and 9 columns each.

Then I am thinking, maybe I should try what is the difference between some `vectorized`  function in R and see how that compares with Rcpp. Again, Rcpp beat apply function, after I changed the record to 1 million rows times 10 columns each row. It took 10+mins and the for loop in R was still running and I have to stop it because I have no idea how long it gonna take.

This experiment totally changed some of my impressions and I started to understand why people really hates for loop in R. Again, all these interesting stories happens in R and I have never jumped out of the R environment, Rcpp makes R possible to it easy to boost the performance of R to C++ level.

If you are using R and found it slow, don’t blame R, blame yourself!

http://sublime-text-unofficial-documentation.readthedocs.org/en/latest/reference/keyboard_shortcuts_osx.html