Profiling the reference implementations

Sam Abbott

2019-12-11

Background

The reference implementation of a Markov model makes use of an array structure and a series of nested loops to sample, simulate and summarise the model. This approach has some clear advantages including making optmisations possible across samples. However, its lack of modularity may make making optimisations more difficult and many also make generalising findings more difficult - as model definitions are linked to model sampling, solving and summarising.

In principle this approach may lead to the most efficient code but this may come at the cost of clarity and generalisability.

Profiling

As expected profiling indicates that the interior iteratively updated for loop operation accounts for the vast majority of the function run time (this makes senses as R is notoriously slow when it comes to this use case). The remaining run-time is taken up by within loop allocation of samples and data summarisation but in comparison to the iterative for loop compute costs are small. Profiling is implemented using the profvis package (Note: Profiling output will be more detailed in an interactive session post package install).

These results suggest several strategies. Firstly, to speed up the within for loop operation (here a matrix multiplication). This would also benefit other resource intensive matrix multiplications. Secondly, to decrease the costs of the iterative for loop itself - perhaps by moving to a Rcpp implementation. Finally, to make use of parallel compute to spread the costs of solving each sample across multiple CPUs.

library(profvis)
# Load SpeedyMarkov from local source.
devtools::load_all("..")
#> Loading SpeedyMarkov

profvis({
  reference_two_state_markov(cycles = 100, samples = 100000)
})