Is Microsoft R Open faster than CRAN R?

Updated 2018-03-25

Motivation

I was finally able to install Microsoft R Open on a machine that features Ubuntu Server 16.04 so now let’s test!

Microsoft© website states that “From its inception, R was designed to use only a single thread (processor) at a time. Even today, R works that way unless linked with multi-threaded BLAS/LAPACK libraries.” That’s true.

But also they state that “Compared to open source R, the MKL offers significant performance gains, particularly on Windows.”. Microsoft R open is, to the date, the best option to install R on Windows in a painless way.

My own results of Microsoft test

This article does not present evidence to close the long discussion on benchmarks nor provides a careful test on different machines.

The benchmarks presented here are an adaptation of Simon Urbanek’s benchmarks, and I’ll be using the version.compare package that is used for the results exposed in Microsoft’s website.

It is important to notice that I did repeat each test 100 times and the results presented here correspond to the median of the repetitions of each test. This was made after a suggestion from George Vega Yon.

Software and hardware specifications

  • System: Ubuntu Server 16.04
  • Processor: Intel© Xeon 2.27GHz (using eight cores)
  • Memory: 32 GB (four DDR3 cards of eight gigabytes each)
  • R version: 3.4.2 different compilations

Results

Performance exhibited running unmodified Microsoft benchmark script (edited R builds location)

Figure 1: Performance exhibited running unmodified Microsoft benchmark script (edited R builds location)

Performance exhibited running modified ATT script (modified to run fresh 100 times and then store the median result)

Figure 2: Performance exhibited running modified ATT script (modified to run fresh 100 times and then store the median result)

Conclusion

These partial results show that CRAN R (aka R Open Source) built with Intel MKL numerical libraries performs better that Microsoft R Open. But, how better? This can’t be extrapolated to other systems but Ubuntu.

For some operations R augmented with alternative math kernel (MKL in this case) performs better than Microsoft R Open.

Appendix: Scripts

I used this script to obtain the results: