Reproducible workflow

As part of a reproducible workflow, caching of function calls, code chunks, and other elements of a project is a critical component. The objective of a reproducible workflow is is likely that an entire work flow from raw data to publication, decision support, report writing, presentation building etc., could be built and be reproducible anywhere, on any computer, operating system, with any starting conditions, on demand. The reproducible::Cache function is built to work with any R function.

Differences with other approaches

Cache uses 2 key the archivist functions saveToLocalRepo and loadFromLocalRepo, but does not use archivist::cache. Similar to archivist::cache, there is some reliance on digest::digest to determine whether the arguments are identical in subsequent iterations; however, it also uses fastdigest::fastdigest to make it substantially faster in many cases. It also but does many things that make standard caching with digest::digest don’t work reliably between systems. For these, the function .robustDigest is introduced to make caching transferable between systems. This is relevant for file paths, environments, parallel clusters, functions (which are contained within an environment), and many others (e.g., see ?.robustDigest for methods). Cache also adds important elements like automated tagging and the option to retrieve disk-cached values via stashed objects in memory using memoise::memoise. This means that running Cache 1, 2, and 3 times on the same function will get progressively faster. This can be extremely useful for web apps built with, say shiny.

Function-level caching

Any function can be cached using: Cache(FUN = functionName, ...).

This will be a slight change to a function call, such as: projectRaster(raster, crs = crs(newRaster)) to Cache(projectRaster, raster, crs = crs(newRaster)).

This is particularly useful for expensive operations.

library(raster)
## Loading required package: sp
library(reproducible)

tmpDir <- file.path(tempdir(), "reproducible_examples", "Cache")
checkPath(tmpDir, create = TRUE)
## [1] "C:/Temp/RtmpAXWdfy/reproducible_examples/Cache"
ras <- raster(extent(0,1000,0,1000), vals = 1:1e6, res = 1)
crs(ras) <- "+proj=lcc +lat_1=48 +lat_2=33 +lon_0=-100 +ellps=WGS84"

newCRS <- "+init=epsg:4326 +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0"
# No Cache
system.time(map1 <- projectRaster(ras, crs = newCRS))
##    user  system elapsed 
##     2.7     0.3     3.0
# With Cache -- a little slower the first time because saving to disk
system.time(map1 <- Cache(projectRaster, ras, crs = newCRS, cacheRepo = tmpDir,
                         notOlderThan = Sys.time()))
##    user  system elapsed 
##    3.60    0.47    4.19
# vastly faster the second time
system.time(map2 <- Cache(projectRaster, ras, crs = newCRS, cacheRepo = tmpDir))
##   loading cached result from previous projectRaster call, adding to memoised copy
##    user  system elapsed 
##    0.11    0.02    0.14
# even faster the third time
system.time(map3 <- Cache(projectRaster, ras, crs = newCRS, cacheRepo = tmpDir))
##   loading memoised result from previous projectRaster call.
##    user  system elapsed 
##    0.05    0.00    0.06
all.equal(map1, map2) # TRUE
## [1] TRUE
all.equal(map1, map3) # TRUE
## [1] TRUE

Caching examples

Basic use

library(raster)
# magrittr, if loaded, gives an error below
  try(detach("package:magrittr", unload = TRUE), silent = TRUE)

try(clearCache(tmpDir), silent = TRUE) # just to make sure it is clear

ranNumsA <- Cache(rnorm, 10, 16, cacheRepo = tmpDir)

# All same
ranNumsB <- Cache(rnorm, 10, 16, cacheRepo = tmpDir) # recovers cached copy
##   loading cached result from previous rnorm call, adding to memoised copy
ranNumsC <- rnorm(10, 16) %>% Cache(cacheRepo = tmpDir) # recovers cached copy
##   loading memoised result from previous 'rnorm' pipe sequence call.
ranNumsD <- Cache(quote(rnorm(n = 10, 16)), cacheRepo = tmpDir) # recovers cached copy
##   loading memoised result from previous rnorm call.
# Any minor change makes it different
ranNumsE <- rnorm(10, 6) %>% Cache(cacheRepo = tmpDir) # different

Example 1: Basic cache use with tags

ranNumsA <- Cache(rnorm, 4, cacheRepo = tmpDir, userTags = "objectName:a")
ranNumsB <- Cache(runif, 4, cacheRepo = tmpDir, userTags = "objectName:b")

showCache(tmpDir, userTags = c("objectName"))
## Cache size:
##   Total (including Rasters): 1 Kb
##   Selected objects (not including Rasters): 512 bytes
##                             artifact         tagKey
##  1: 072ed8d215e14bfc63d3e9e73376e0d8         format
##  2: 072ed8d215e14bfc63d3e9e73376e0d8           name
##  3: 072ed8d215e14bfc63d3e9e73376e0d8          class
##  4: 072ed8d215e14bfc63d3e9e73376e0d8           date
##  5: 072ed8d215e14bfc63d3e9e73376e0d8        cacheId
##  6: 072ed8d215e14bfc63d3e9e73376e0d8     objectName
##  7: 072ed8d215e14bfc63d3e9e73376e0d8       function
##  8: 072ed8d215e14bfc63d3e9e73376e0d8    object.size
##  9: 072ed8d215e14bfc63d3e9e73376e0d8       accessed
## 10: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 11: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 12: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 13: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 14: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 15: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 16: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 17: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 18: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 19: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 20: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 21: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 22: 072ed8d215e14bfc63d3e9e73376e0d8      preDigest
## 23: 072ed8d215e14bfc63d3e9e73376e0d8      preDigest
## 24: 724fd3ff5fc0573722a514d5e7065b6e         format
## 25: 724fd3ff5fc0573722a514d5e7065b6e           name
## 26: 724fd3ff5fc0573722a514d5e7065b6e          class
## 27: 724fd3ff5fc0573722a514d5e7065b6e           date
## 28: 724fd3ff5fc0573722a514d5e7065b6e        cacheId
## 29: 724fd3ff5fc0573722a514d5e7065b6e     objectName
## 30: 724fd3ff5fc0573722a514d5e7065b6e       function
## 31: 724fd3ff5fc0573722a514d5e7065b6e    object.size
## 32: 724fd3ff5fc0573722a514d5e7065b6e       accessed
## 33: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 34: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 35: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 36: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 37: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 38: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 39: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 40: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 41: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 42: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 43: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 44: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 45: 724fd3ff5fc0573722a514d5e7065b6e      preDigest
## 46: 724fd3ff5fc0573722a514d5e7065b6e      preDigest
##                             artifact         tagKey
##                                  tagValue         createdDate
##  1:                                   rda 2018-09-11 17:26:59
##  2:      072ed8d215e14bfc63d3e9e73376e0d8 2018-09-11 17:26:59
##  3:                               numeric 2018-09-11 17:26:59
##  4:                   2018-09-11 17:26:59 2018-09-11 17:26:59
##  5:      f55d44974d91fbf61b6cb524983b1ebc 2018-09-11 17:26:59
##  6:                                     a 2018-09-11 17:26:59
##  7:                                 rnorm 2018-09-11 17:26:59
##  8:                                  1024 2018-09-11 17:26:59
##  9:                   2018-09-11 17:26:59 2018-09-11 17:26:59
## 10:                   withCallingHandlers 2018-09-11 17:26:59
## 11:                               saveRDS 2018-09-11 17:26:59
## 12:                               do.call 2018-09-11 17:26:59
## 13:                          process_file 2018-09-11 17:26:59
## 14:                         process_group 2018-09-11 17:26:59
## 15:                   process_group.block 2018-09-11 17:26:59
## 16:                            call_block 2018-09-11 17:26:59
## 17:                            block_exec 2018-09-11 17:26:59
## 18:                                in_dir 2018-09-11 17:26:59
## 19:                             timing_fn 2018-09-11 17:26:59
## 20:                                handle 2018-09-11 17:26:59
## 21:                           withVisible 2018-09-11 17:26:59
## 22:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:26:59
## 23: .FUN:a979d0700a6ef8c7c18736096ff1d522 2018-09-11 17:26:59
## 24:                                   rda 2018-09-11 17:27:00
## 25:      724fd3ff5fc0573722a514d5e7065b6e 2018-09-11 17:27:00
## 26:                               numeric 2018-09-11 17:27:00
## 27:                   2018-09-11 17:27:00 2018-09-11 17:27:00
## 28:      7cb09aa387ba235db013863648303b34 2018-09-11 17:27:00
## 29:                                     b 2018-09-11 17:27:00
## 30:                                 runif 2018-09-11 17:27:00
## 31:                                  1024 2018-09-11 17:27:00
## 32:                   2018-09-11 17:27:00 2018-09-11 17:27:00
## 33:                   withCallingHandlers 2018-09-11 17:27:00
## 34:                               saveRDS 2018-09-11 17:27:00
## 35:                               do.call 2018-09-11 17:27:00
## 36:                          process_file 2018-09-11 17:27:00
## 37:                         process_group 2018-09-11 17:27:00
## 38:                   process_group.block 2018-09-11 17:27:00
## 39:                            call_block 2018-09-11 17:27:00
## 40:                            block_exec 2018-09-11 17:27:00
## 41:                                in_dir 2018-09-11 17:27:00
## 42:                             timing_fn 2018-09-11 17:27:00
## 43:                                handle 2018-09-11 17:27:00
## 44:                           withVisible 2018-09-11 17:27:00
## 45:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:27:00
## 46: .FUN:179092fbad3ec261e58bc89bc8a703c3 2018-09-11 17:27:00
##                                  tagValue         createdDate
showCache(tmpDir, userTags = c("^a$")) # regular expression ... "a" exactly
## Cache size:
##   Total (including Rasters): 1 Kb
##   Selected objects (not including Rasters): 256 bytes
##                             artifact         tagKey
##  1: 072ed8d215e14bfc63d3e9e73376e0d8         format
##  2: 072ed8d215e14bfc63d3e9e73376e0d8           name
##  3: 072ed8d215e14bfc63d3e9e73376e0d8          class
##  4: 072ed8d215e14bfc63d3e9e73376e0d8           date
##  5: 072ed8d215e14bfc63d3e9e73376e0d8        cacheId
##  6: 072ed8d215e14bfc63d3e9e73376e0d8     objectName
##  7: 072ed8d215e14bfc63d3e9e73376e0d8       function
##  8: 072ed8d215e14bfc63d3e9e73376e0d8    object.size
##  9: 072ed8d215e14bfc63d3e9e73376e0d8       accessed
## 10: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 11: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 12: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 13: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 14: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 15: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 16: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 17: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 18: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 19: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 20: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 21: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 22: 072ed8d215e14bfc63d3e9e73376e0d8      preDigest
## 23: 072ed8d215e14bfc63d3e9e73376e0d8      preDigest
##                             artifact         tagKey
##                                  tagValue         createdDate
##  1:                                   rda 2018-09-11 17:26:59
##  2:      072ed8d215e14bfc63d3e9e73376e0d8 2018-09-11 17:26:59
##  3:                               numeric 2018-09-11 17:26:59
##  4:                   2018-09-11 17:26:59 2018-09-11 17:26:59
##  5:      f55d44974d91fbf61b6cb524983b1ebc 2018-09-11 17:26:59
##  6:                                     a 2018-09-11 17:26:59
##  7:                                 rnorm 2018-09-11 17:26:59
##  8:                                  1024 2018-09-11 17:26:59
##  9:                   2018-09-11 17:26:59 2018-09-11 17:26:59
## 10:                   withCallingHandlers 2018-09-11 17:26:59
## 11:                               saveRDS 2018-09-11 17:26:59
## 12:                               do.call 2018-09-11 17:26:59
## 13:                          process_file 2018-09-11 17:26:59
## 14:                         process_group 2018-09-11 17:26:59
## 15:                   process_group.block 2018-09-11 17:26:59
## 16:                            call_block 2018-09-11 17:26:59
## 17:                            block_exec 2018-09-11 17:26:59
## 18:                                in_dir 2018-09-11 17:26:59
## 19:                             timing_fn 2018-09-11 17:26:59
## 20:                                handle 2018-09-11 17:26:59
## 21:                           withVisible 2018-09-11 17:26:59
## 22:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:26:59
## 23: .FUN:a979d0700a6ef8c7c18736096ff1d522 2018-09-11 17:26:59
##                                  tagValue         createdDate
showCache(tmpDir, userTags = c("runif")) # show only cached objects made during runif call
## Cache size:
##   Total (including Rasters): 1 Kb
##   Selected objects (not including Rasters): 256 bytes
##                             artifact         tagKey
##  1: 724fd3ff5fc0573722a514d5e7065b6e         format
##  2: 724fd3ff5fc0573722a514d5e7065b6e           name
##  3: 724fd3ff5fc0573722a514d5e7065b6e          class
##  4: 724fd3ff5fc0573722a514d5e7065b6e           date
##  5: 724fd3ff5fc0573722a514d5e7065b6e        cacheId
##  6: 724fd3ff5fc0573722a514d5e7065b6e     objectName
##  7: 724fd3ff5fc0573722a514d5e7065b6e       function
##  8: 724fd3ff5fc0573722a514d5e7065b6e    object.size
##  9: 724fd3ff5fc0573722a514d5e7065b6e       accessed
## 10: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 11: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 12: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 13: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 14: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 15: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 16: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 17: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 18: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 19: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 20: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 21: 724fd3ff5fc0573722a514d5e7065b6e otherFunctions
## 22: 724fd3ff5fc0573722a514d5e7065b6e      preDigest
## 23: 724fd3ff5fc0573722a514d5e7065b6e      preDigest
##                             artifact         tagKey
##                                  tagValue         createdDate
##  1:                                   rda 2018-09-11 17:27:00
##  2:      724fd3ff5fc0573722a514d5e7065b6e 2018-09-11 17:27:00
##  3:                               numeric 2018-09-11 17:27:00
##  4:                   2018-09-11 17:27:00 2018-09-11 17:27:00
##  5:      7cb09aa387ba235db013863648303b34 2018-09-11 17:27:00
##  6:                                     b 2018-09-11 17:27:00
##  7:                                 runif 2018-09-11 17:27:00
##  8:                                  1024 2018-09-11 17:27:00
##  9:                   2018-09-11 17:27:00 2018-09-11 17:27:00
## 10:                   withCallingHandlers 2018-09-11 17:27:00
## 11:                               saveRDS 2018-09-11 17:27:00
## 12:                               do.call 2018-09-11 17:27:00
## 13:                          process_file 2018-09-11 17:27:00
## 14:                         process_group 2018-09-11 17:27:00
## 15:                   process_group.block 2018-09-11 17:27:00
## 16:                            call_block 2018-09-11 17:27:00
## 17:                            block_exec 2018-09-11 17:27:00
## 18:                                in_dir 2018-09-11 17:27:00
## 19:                             timing_fn 2018-09-11 17:27:00
## 20:                                handle 2018-09-11 17:27:00
## 21:                           withVisible 2018-09-11 17:27:00
## 22:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:27:00
## 23: .FUN:179092fbad3ec261e58bc89bc8a703c3 2018-09-11 17:27:00
##                                  tagValue         createdDate
clearCache(tmpDir, userTags = c("runif")) # remove only cached objects made during runif call
## Cache size:
##   Total (including Rasters): 1 Kb
##   Selected objects (not including Rasters): 256 bytes
showCache(tmpDir) # only those made during rnorm call
## Cache size:
##   Total (including Rasters): 816 bytes
##   Selected objects (not including Rasters): 816 bytes
##                             artifact         tagKey
##  1: 072ed8d215e14bfc63d3e9e73376e0d8         format
##  2: 072ed8d215e14bfc63d3e9e73376e0d8           name
##  3: 072ed8d215e14bfc63d3e9e73376e0d8          class
##  4: 072ed8d215e14bfc63d3e9e73376e0d8           date
##  5: 072ed8d215e14bfc63d3e9e73376e0d8        cacheId
##  6: 072ed8d215e14bfc63d3e9e73376e0d8     objectName
##  7: 072ed8d215e14bfc63d3e9e73376e0d8       function
##  8: 072ed8d215e14bfc63d3e9e73376e0d8    object.size
##  9: 072ed8d215e14bfc63d3e9e73376e0d8       accessed
## 10: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 11: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 12: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 13: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 14: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 15: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 16: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 17: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 18: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 19: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 20: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 21: 072ed8d215e14bfc63d3e9e73376e0d8 otherFunctions
## 22: 072ed8d215e14bfc63d3e9e73376e0d8      preDigest
## 23: 072ed8d215e14bfc63d3e9e73376e0d8      preDigest
## 24: 3418586fe39f5d5c4dc458f141ae0e99         format
## 25: 3418586fe39f5d5c4dc458f141ae0e99           name
## 26: 3418586fe39f5d5c4dc458f141ae0e99          class
## 27: 3418586fe39f5d5c4dc458f141ae0e99           date
## 28: 3418586fe39f5d5c4dc458f141ae0e99        cacheId
## 29: 3418586fe39f5d5c4dc458f141ae0e99       function
## 30: 3418586fe39f5d5c4dc458f141ae0e99    object.size
## 31: 3418586fe39f5d5c4dc458f141ae0e99       accessed
## 32: 3418586fe39f5d5c4dc458f141ae0e99 otherFunctions
## 33: 3418586fe39f5d5c4dc458f141ae0e99 otherFunctions
## 34: 3418586fe39f5d5c4dc458f141ae0e99 otherFunctions
## 35: 3418586fe39f5d5c4dc458f141ae0e99 otherFunctions
## 36: 3418586fe39f5d5c4dc458f141ae0e99 otherFunctions
## 37: 3418586fe39f5d5c4dc458f141ae0e99 otherFunctions
## 38: 3418586fe39f5d5c4dc458f141ae0e99 otherFunctions
## 39: 3418586fe39f5d5c4dc458f141ae0e99 otherFunctions
## 40: 3418586fe39f5d5c4dc458f141ae0e99 otherFunctions
## 41: 3418586fe39f5d5c4dc458f141ae0e99 otherFunctions
## 42: 3418586fe39f5d5c4dc458f141ae0e99 otherFunctions
## 43: 3418586fe39f5d5c4dc458f141ae0e99 otherFunctions
## 44: 3418586fe39f5d5c4dc458f141ae0e99      preDigest
## 45: 3418586fe39f5d5c4dc458f141ae0e99      preDigest
## 46: 3418586fe39f5d5c4dc458f141ae0e99      preDigest
## 47: 3418586fe39f5d5c4dc458f141ae0e99       accessed
## 48: 3418586fe39f5d5c4dc458f141ae0e99       accessed
## 49: 3418586fe39f5d5c4dc458f141ae0e99       accessed
## 50: eb59561785d6d945627c533d9b6e82ab         format
## 51: eb59561785d6d945627c533d9b6e82ab           name
## 52: eb59561785d6d945627c533d9b6e82ab          class
## 53: eb59561785d6d945627c533d9b6e82ab           date
## 54: eb59561785d6d945627c533d9b6e82ab        cacheId
## 55: eb59561785d6d945627c533d9b6e82ab       function
## 56: eb59561785d6d945627c533d9b6e82ab    object.size
## 57: eb59561785d6d945627c533d9b6e82ab       accessed
## 58: eb59561785d6d945627c533d9b6e82ab otherFunctions
## 59: eb59561785d6d945627c533d9b6e82ab otherFunctions
## 60: eb59561785d6d945627c533d9b6e82ab otherFunctions
## 61: eb59561785d6d945627c533d9b6e82ab otherFunctions
## 62: eb59561785d6d945627c533d9b6e82ab otherFunctions
## 63: eb59561785d6d945627c533d9b6e82ab otherFunctions
## 64: eb59561785d6d945627c533d9b6e82ab otherFunctions
## 65: eb59561785d6d945627c533d9b6e82ab otherFunctions
## 66: eb59561785d6d945627c533d9b6e82ab otherFunctions
## 67: eb59561785d6d945627c533d9b6e82ab otherFunctions
## 68: eb59561785d6d945627c533d9b6e82ab otherFunctions
## 69: eb59561785d6d945627c533d9b6e82ab otherFunctions
## 70: eb59561785d6d945627c533d9b6e82ab otherFunctions
## 71: eb59561785d6d945627c533d9b6e82ab      preDigest
## 72: eb59561785d6d945627c533d9b6e82ab      preDigest
## 73: eb59561785d6d945627c533d9b6e82ab      preDigest
##                             artifact         tagKey
##                                  tagValue         createdDate
##  1:                                   rda 2018-09-11 17:26:59
##  2:      072ed8d215e14bfc63d3e9e73376e0d8 2018-09-11 17:26:59
##  3:                               numeric 2018-09-11 17:26:59
##  4:                   2018-09-11 17:26:59 2018-09-11 17:26:59
##  5:      f55d44974d91fbf61b6cb524983b1ebc 2018-09-11 17:26:59
##  6:                                     a 2018-09-11 17:26:59
##  7:                                 rnorm 2018-09-11 17:26:59
##  8:                                  1024 2018-09-11 17:26:59
##  9:                   2018-09-11 17:26:59 2018-09-11 17:26:59
## 10:                   withCallingHandlers 2018-09-11 17:26:59
## 11:                               saveRDS 2018-09-11 17:26:59
## 12:                               do.call 2018-09-11 17:26:59
## 13:                          process_file 2018-09-11 17:26:59
## 14:                         process_group 2018-09-11 17:26:59
## 15:                   process_group.block 2018-09-11 17:26:59
## 16:                            call_block 2018-09-11 17:26:59
## 17:                            block_exec 2018-09-11 17:26:59
## 18:                                in_dir 2018-09-11 17:26:59
## 19:                             timing_fn 2018-09-11 17:26:59
## 20:                                handle 2018-09-11 17:26:59
## 21:                           withVisible 2018-09-11 17:26:59
## 22:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:26:59
## 23: .FUN:a979d0700a6ef8c7c18736096ff1d522 2018-09-11 17:26:59
## 24:                                   rda 2018-09-11 17:26:58
## 25:      3418586fe39f5d5c4dc458f141ae0e99 2018-09-11 17:26:58
## 26:                               numeric 2018-09-11 17:26:58
## 27:                   2018-09-11 17:26:58 2018-09-11 17:26:58
## 28:      c878d1fb026c0854fc0480553fa4cf24 2018-09-11 17:26:58
## 29:                                 rnorm 2018-09-11 17:26:58
## 30:                                  1120 2018-09-11 17:26:58
## 31:                   2018-09-11 17:26:58 2018-09-11 17:26:58
## 32:                   withCallingHandlers 2018-09-11 17:26:58
## 33:                               saveRDS 2018-09-11 17:26:58
## 34:                               do.call 2018-09-11 17:26:58
## 35:                          process_file 2018-09-11 17:26:58
## 36:                         process_group 2018-09-11 17:26:58
## 37:                   process_group.block 2018-09-11 17:26:58
## 38:                            call_block 2018-09-11 17:26:58
## 39:                            block_exec 2018-09-11 17:26:58
## 40:                                in_dir 2018-09-11 17:26:58
## 41:                             timing_fn 2018-09-11 17:26:58
## 42:                                handle 2018-09-11 17:26:58
## 43:                           withVisible 2018-09-11 17:26:58
## 44:    n:c192d58bc0a9ddf2b5f916d929bccc08 2018-09-11 17:26:59
## 45: mean:eab1c55b26658701ccc6e12b858d29e4 2018-09-11 17:26:59
## 46: .FUN:a979d0700a6ef8c7c18736096ff1d522 2018-09-11 17:26:59
## 47:                   2018-09-11 17:26:59 2018-09-11 17:26:59
## 48:                   2018-09-11 17:26:59 2018-09-11 17:26:59
## 49:                   2018-09-11 17:26:59 2018-09-11 17:26:59
## 50:                                   rda 2018-09-11 17:26:59
## 51:      eb59561785d6d945627c533d9b6e82ab 2018-09-11 17:26:59
## 52:                               numeric 2018-09-11 17:26:59
## 53:                   2018-09-11 17:26:59 2018-09-11 17:26:59
## 54:      93a7b68b228b551669eb36211e857b39 2018-09-11 17:26:59
## 55:                   rnorm pipe sequence 2018-09-11 17:26:59
## 56:                                  1120 2018-09-11 17:26:59
## 57:                   2018-09-11 17:26:59 2018-09-11 17:26:59
## 58:                   withCallingHandlers 2018-09-11 17:26:59
## 59:                               saveRDS 2018-09-11 17:26:59
## 60:                               do.call 2018-09-11 17:26:59
## 61:                          process_file 2018-09-11 17:26:59
## 62:                         process_group 2018-09-11 17:26:59
## 63:                   process_group.block 2018-09-11 17:26:59
## 64:                            call_block 2018-09-11 17:26:59
## 65:                            block_exec 2018-09-11 17:26:59
## 66:                                in_dir 2018-09-11 17:26:59
## 67:                             timing_fn 2018-09-11 17:26:59
## 68:                                handle 2018-09-11 17:26:59
## 69:                           withVisible 2018-09-11 17:26:59
## 70:                                   %>% 2018-09-11 17:26:59
## 71:    n:c192d58bc0a9ddf2b5f916d929bccc08 2018-09-11 17:26:59
## 72: mean:55835193261a0d71e70d137dfa31cd5b 2018-09-11 17:26:59
## 73: .FUN:a979d0700a6ef8c7c18736096ff1d522 2018-09-11 17:26:59
##                                  tagValue         createdDate
clearCache(tmpDir)

Example 2: using the “accessed” tag

ranNumsA <- Cache(rnorm, 4, cacheRepo = tmpDir, userTags = "objectName:a")
ranNumsB <- Cache(runif, 4, cacheRepo = tmpDir, userTags = "objectName:b")

# access it again, from Cache
ranNumsA <- Cache(rnorm, 4, cacheRepo = tmpDir, userTags = "objectName:a")
##   loading cached result from previous rnorm call, adding to memoised copy
wholeCache <- showCache(tmpDir)
## Cache size:
##   Total (including Rasters): 512 bytes
##   Selected objects (not including Rasters): 512 bytes
# keep only items accessed "recently" (i.e., only objectName:a)
onlyRecentlyAccessed <- showCache(tmpDir, userTags = max(wholeCache[tagKey == "accessed"]$tagValue))
## Cache size:
##   Total (including Rasters): 512 bytes
##   Selected objects (not including Rasters): 512 bytes
# inverse join with 2 data.tables ... using: a[!b]
# i.e., return all of wholeCache that was not recently accessed
toRemove <- unique(wholeCache[!onlyRecentlyAccessed], by = "artifact")$artifact
clearCache(tmpDir, toRemove) # remove ones not recently accessed
## Cache size:
##   Total (including Rasters): 512 bytes
##   Selected objects (not including Rasters): 512 bytes
showCache(tmpDir) # still has more recently accessed
## Cache size:
##   Total (including Rasters): 0 bytes
##   Selected objects (not including Rasters): 0 bytes
## Empty data.table (0 rows) of 3 cols: md5hash,name,createdDate
clearCache(tmpDir)

Example 3: using keepCache

ranNumsA <- Cache(rnorm, 4, cacheRepo = tmpDir, userTags = "objectName:a")
ranNumsB <- Cache(runif, 4, cacheRepo = tmpDir, userTags = "objectName:b")

# keep only those cached items from the last 24 hours
oneDay <- 60 * 60 * 24
keepCache(tmpDir, after = Sys.time() - oneDay)
## Cache size:
##   Total (including Rasters): 512 bytes
##   Selected objects (not including Rasters): 512 bytes
##                             artifact         tagKey
##  1: 1560d2916eb5eccf63c59d6300f90641         format
##  2: 1560d2916eb5eccf63c59d6300f90641           name
##  3: 1560d2916eb5eccf63c59d6300f90641          class
##  4: 1560d2916eb5eccf63c59d6300f90641           date
##  5: 1560d2916eb5eccf63c59d6300f90641        cacheId
##  6: 1560d2916eb5eccf63c59d6300f90641     objectName
##  7: 1560d2916eb5eccf63c59d6300f90641       function
##  8: 1560d2916eb5eccf63c59d6300f90641    object.size
##  9: 1560d2916eb5eccf63c59d6300f90641       accessed
## 10: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 11: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 12: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 13: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 14: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 15: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 16: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 17: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 18: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 19: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 20: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 21: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 22: 1560d2916eb5eccf63c59d6300f90641      preDigest
## 23: 1560d2916eb5eccf63c59d6300f90641      preDigest
## 24: 7bb08a0ce8fbea954de1167bec83eebd         format
## 25: 7bb08a0ce8fbea954de1167bec83eebd           name
## 26: 7bb08a0ce8fbea954de1167bec83eebd          class
## 27: 7bb08a0ce8fbea954de1167bec83eebd           date
## 28: 7bb08a0ce8fbea954de1167bec83eebd        cacheId
## 29: 7bb08a0ce8fbea954de1167bec83eebd     objectName
## 30: 7bb08a0ce8fbea954de1167bec83eebd       function
## 31: 7bb08a0ce8fbea954de1167bec83eebd    object.size
## 32: 7bb08a0ce8fbea954de1167bec83eebd       accessed
## 33: 7bb08a0ce8fbea954de1167bec83eebd otherFunctions
## 34: 7bb08a0ce8fbea954de1167bec83eebd otherFunctions
## 35: 7bb08a0ce8fbea954de1167bec83eebd otherFunctions
## 36: 7bb08a0ce8fbea954de1167bec83eebd otherFunctions
## 37: 7bb08a0ce8fbea954de1167bec83eebd otherFunctions
## 38: 7bb08a0ce8fbea954de1167bec83eebd otherFunctions
## 39: 7bb08a0ce8fbea954de1167bec83eebd otherFunctions
## 40: 7bb08a0ce8fbea954de1167bec83eebd otherFunctions
## 41: 7bb08a0ce8fbea954de1167bec83eebd otherFunctions
## 42: 7bb08a0ce8fbea954de1167bec83eebd otherFunctions
## 43: 7bb08a0ce8fbea954de1167bec83eebd otherFunctions
## 44: 7bb08a0ce8fbea954de1167bec83eebd otherFunctions
## 45: 7bb08a0ce8fbea954de1167bec83eebd      preDigest
## 46: 7bb08a0ce8fbea954de1167bec83eebd      preDigest
##                             artifact         tagKey
##                                  tagValue         createdDate
##  1:                                   rda 2018-09-11 17:27:01
##  2:      1560d2916eb5eccf63c59d6300f90641 2018-09-11 17:27:01
##  3:                               numeric 2018-09-11 17:27:01
##  4:                   2018-09-11 17:27:01 2018-09-11 17:27:01
##  5:      f55d44974d91fbf61b6cb524983b1ebc 2018-09-11 17:27:01
##  6:                                     a 2018-09-11 17:27:01
##  7:                                 rnorm 2018-09-11 17:27:01
##  8:                                  1024 2018-09-11 17:27:01
##  9:                   2018-09-11 17:27:01 2018-09-11 17:27:01
## 10:                   withCallingHandlers 2018-09-11 17:27:01
## 11:                               saveRDS 2018-09-11 17:27:01
## 12:                               do.call 2018-09-11 17:27:01
## 13:                          process_file 2018-09-11 17:27:01
## 14:                         process_group 2018-09-11 17:27:01
## 15:                   process_group.block 2018-09-11 17:27:01
## 16:                            call_block 2018-09-11 17:27:01
## 17:                            block_exec 2018-09-11 17:27:01
## 18:                                in_dir 2018-09-11 17:27:01
## 19:                             timing_fn 2018-09-11 17:27:01
## 20:                                handle 2018-09-11 17:27:01
## 21:                           withVisible 2018-09-11 17:27:01
## 22:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:27:01
## 23: .FUN:a979d0700a6ef8c7c18736096ff1d522 2018-09-11 17:27:01
## 24:                                   rda 2018-09-11 17:27:02
## 25:      7bb08a0ce8fbea954de1167bec83eebd 2018-09-11 17:27:02
## 26:                               numeric 2018-09-11 17:27:02
## 27:                   2018-09-11 17:27:02 2018-09-11 17:27:02
## 28:      7cb09aa387ba235db013863648303b34 2018-09-11 17:27:02
## 29:                                     b 2018-09-11 17:27:02
## 30:                                 runif 2018-09-11 17:27:02
## 31:                                  1024 2018-09-11 17:27:02
## 32:                   2018-09-11 17:27:01 2018-09-11 17:27:02
## 33:                   withCallingHandlers 2018-09-11 17:27:02
## 34:                               saveRDS 2018-09-11 17:27:02
## 35:                               do.call 2018-09-11 17:27:02
## 36:                          process_file 2018-09-11 17:27:02
## 37:                         process_group 2018-09-11 17:27:02
## 38:                   process_group.block 2018-09-11 17:27:02
## 39:                            call_block 2018-09-11 17:27:02
## 40:                            block_exec 2018-09-11 17:27:02
## 41:                                in_dir 2018-09-11 17:27:02
## 42:                             timing_fn 2018-09-11 17:27:02
## 43:                                handle 2018-09-11 17:27:02
## 44:                           withVisible 2018-09-11 17:27:02
## 45:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:27:02
## 46: .FUN:179092fbad3ec261e58bc89bc8a703c3 2018-09-11 17:27:02
##                                  tagValue         createdDate
# Keep all Cache items created with an rnorm() call
keepCache(tmpDir, userTags = "rnorm")
## Cache size:
##   Total (including Rasters): 512 bytes
##   Selected objects (not including Rasters): 256 bytes
##                             artifact         tagKey
##  1: 1560d2916eb5eccf63c59d6300f90641         format
##  2: 1560d2916eb5eccf63c59d6300f90641           name
##  3: 1560d2916eb5eccf63c59d6300f90641          class
##  4: 1560d2916eb5eccf63c59d6300f90641           date
##  5: 1560d2916eb5eccf63c59d6300f90641        cacheId
##  6: 1560d2916eb5eccf63c59d6300f90641     objectName
##  7: 1560d2916eb5eccf63c59d6300f90641       function
##  8: 1560d2916eb5eccf63c59d6300f90641    object.size
##  9: 1560d2916eb5eccf63c59d6300f90641       accessed
## 10: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 11: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 12: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 13: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 14: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 15: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 16: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 17: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 18: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 19: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 20: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 21: 1560d2916eb5eccf63c59d6300f90641 otherFunctions
## 22: 1560d2916eb5eccf63c59d6300f90641      preDigest
## 23: 1560d2916eb5eccf63c59d6300f90641      preDigest
##                             artifact         tagKey
##                                  tagValue         createdDate
##  1:                                   rda 2018-09-11 17:27:01
##  2:      1560d2916eb5eccf63c59d6300f90641 2018-09-11 17:27:01
##  3:                               numeric 2018-09-11 17:27:01
##  4:                   2018-09-11 17:27:01 2018-09-11 17:27:01
##  5:      f55d44974d91fbf61b6cb524983b1ebc 2018-09-11 17:27:01
##  6:                                     a 2018-09-11 17:27:01
##  7:                                 rnorm 2018-09-11 17:27:01
##  8:                                  1024 2018-09-11 17:27:01
##  9:                   2018-09-11 17:27:01 2018-09-11 17:27:01
## 10:                   withCallingHandlers 2018-09-11 17:27:01
## 11:                               saveRDS 2018-09-11 17:27:01
## 12:                               do.call 2018-09-11 17:27:01
## 13:                          process_file 2018-09-11 17:27:01
## 14:                         process_group 2018-09-11 17:27:01
## 15:                   process_group.block 2018-09-11 17:27:01
## 16:                            call_block 2018-09-11 17:27:01
## 17:                            block_exec 2018-09-11 17:27:01
## 18:                                in_dir 2018-09-11 17:27:01
## 19:                             timing_fn 2018-09-11 17:27:01
## 20:                                handle 2018-09-11 17:27:01
## 21:                           withVisible 2018-09-11 17:27:01
## 22:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:27:01
## 23: .FUN:a979d0700a6ef8c7c18736096ff1d522 2018-09-11 17:27:01
##                                  tagValue         createdDate
# Remove all Cache items that happened within a rnorm() call
clearCache(tmpDir, userTags = "rnorm")
## Cache size:
##   Total (including Rasters): 256 bytes
##   Selected objects (not including Rasters): 256 bytes
showCache(tmpDir) ## empty
## Cache size:
##   Total (including Rasters): 0 bytes
##   Selected objects (not including Rasters): 0 bytes
## Empty data.table (0 rows) of 3 cols: md5hash,name,createdDate

Example 4: searching for multiple objects in the cache

# default userTags is "and" matching; for "or" matching use |
ranNumsA <- Cache(runif, 4, cacheRepo = tmpDir, userTags = "objectName:a")
ranNumsB <- Cache(rnorm, 4, cacheRepo = tmpDir, userTags = "objectName:b")

# show all objects (runif and rnorm in this case)
showCache(tmpDir)
## Cache size:
##   Total (including Rasters): 512 bytes
##   Selected objects (not including Rasters): 512 bytes
##                             artifact         tagKey
##  1: 556c3fbd14f8cdcf8bd19227136aa3bb         format
##  2: 556c3fbd14f8cdcf8bd19227136aa3bb           name
##  3: 556c3fbd14f8cdcf8bd19227136aa3bb          class
##  4: 556c3fbd14f8cdcf8bd19227136aa3bb           date
##  5: 556c3fbd14f8cdcf8bd19227136aa3bb        cacheId
##  6: 556c3fbd14f8cdcf8bd19227136aa3bb     objectName
##  7: 556c3fbd14f8cdcf8bd19227136aa3bb       function
##  8: 556c3fbd14f8cdcf8bd19227136aa3bb    object.size
##  9: 556c3fbd14f8cdcf8bd19227136aa3bb       accessed
## 10: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 11: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 12: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 13: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 14: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 15: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 16: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 17: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 18: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 19: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 20: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 21: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 22: 556c3fbd14f8cdcf8bd19227136aa3bb      preDigest
## 23: 556c3fbd14f8cdcf8bd19227136aa3bb      preDigest
## 24: 7450fdaf3a485327278cec0e06f7472e         format
## 25: 7450fdaf3a485327278cec0e06f7472e           name
## 26: 7450fdaf3a485327278cec0e06f7472e          class
## 27: 7450fdaf3a485327278cec0e06f7472e           date
## 28: 7450fdaf3a485327278cec0e06f7472e        cacheId
## 29: 7450fdaf3a485327278cec0e06f7472e     objectName
## 30: 7450fdaf3a485327278cec0e06f7472e       function
## 31: 7450fdaf3a485327278cec0e06f7472e    object.size
## 32: 7450fdaf3a485327278cec0e06f7472e       accessed
## 33: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 34: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 35: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 36: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 37: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 38: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 39: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 40: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 41: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 42: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 43: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 44: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 45: 7450fdaf3a485327278cec0e06f7472e      preDigest
## 46: 7450fdaf3a485327278cec0e06f7472e      preDigest
##                             artifact         tagKey
##                                  tagValue         createdDate
##  1:                                   rda 2018-09-11 17:27:02
##  2:      556c3fbd14f8cdcf8bd19227136aa3bb 2018-09-11 17:27:02
##  3:                               numeric 2018-09-11 17:27:02
##  4:                   2018-09-11 17:27:02 2018-09-11 17:27:03
##  5:      f55d44974d91fbf61b6cb524983b1ebc 2018-09-11 17:27:03
##  6:                                     b 2018-09-11 17:27:03
##  7:                                 rnorm 2018-09-11 17:27:03
##  8:                                  1024 2018-09-11 17:27:03
##  9:                   2018-09-11 17:27:02 2018-09-11 17:27:03
## 10:                   withCallingHandlers 2018-09-11 17:27:03
## 11:                               saveRDS 2018-09-11 17:27:03
## 12:                               do.call 2018-09-11 17:27:03
## 13:                          process_file 2018-09-11 17:27:03
## 14:                         process_group 2018-09-11 17:27:03
## 15:                   process_group.block 2018-09-11 17:27:03
## 16:                            call_block 2018-09-11 17:27:03
## 17:                            block_exec 2018-09-11 17:27:03
## 18:                                in_dir 2018-09-11 17:27:03
## 19:                             timing_fn 2018-09-11 17:27:03
## 20:                                handle 2018-09-11 17:27:03
## 21:                           withVisible 2018-09-11 17:27:03
## 22:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:27:03
## 23: .FUN:a979d0700a6ef8c7c18736096ff1d522 2018-09-11 17:27:03
## 24:                                   rda 2018-09-11 17:27:02
## 25:      7450fdaf3a485327278cec0e06f7472e 2018-09-11 17:27:02
## 26:                               numeric 2018-09-11 17:27:02
## 27:                   2018-09-11 17:27:02 2018-09-11 17:27:02
## 28:      7cb09aa387ba235db013863648303b34 2018-09-11 17:27:02
## 29:                                     a 2018-09-11 17:27:02
## 30:                                 runif 2018-09-11 17:27:02
## 31:                                  1024 2018-09-11 17:27:02
## 32:                   2018-09-11 17:27:02 2018-09-11 17:27:02
## 33:                   withCallingHandlers 2018-09-11 17:27:02
## 34:                               saveRDS 2018-09-11 17:27:02
## 35:                               do.call 2018-09-11 17:27:02
## 36:                          process_file 2018-09-11 17:27:02
## 37:                         process_group 2018-09-11 17:27:02
## 38:                   process_group.block 2018-09-11 17:27:02
## 39:                            call_block 2018-09-11 17:27:02
## 40:                            block_exec 2018-09-11 17:27:02
## 41:                                in_dir 2018-09-11 17:27:02
## 42:                             timing_fn 2018-09-11 17:27:02
## 43:                                handle 2018-09-11 17:27:02
## 44:                           withVisible 2018-09-11 17:27:02
## 45:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:27:02
## 46: .FUN:179092fbad3ec261e58bc89bc8a703c3 2018-09-11 17:27:02
##                                  tagValue         createdDate
# show objects that are both runif and rnorm
# (i.e., none in this case, because objecs are either or, not both)
showCache(tmpDir, userTags = c("runif", "rnorm")) ## empty
## Cache size:
##   Total (including Rasters): 512 bytes
##   Selected objects (not including Rasters): 0 bytes
## Empty data.table (0 rows) of 4 cols: artifact,tagKey,tagValue,createdDate
# show objects that are either runif or rnorm ("or" search)
showCache(tmpDir, userTags = "runif|rnorm")
## Cache size:
##   Total (including Rasters): 512 bytes
##   Selected objects (not including Rasters): 512 bytes
##                             artifact         tagKey
##  1: 556c3fbd14f8cdcf8bd19227136aa3bb         format
##  2: 556c3fbd14f8cdcf8bd19227136aa3bb           name
##  3: 556c3fbd14f8cdcf8bd19227136aa3bb          class
##  4: 556c3fbd14f8cdcf8bd19227136aa3bb           date
##  5: 556c3fbd14f8cdcf8bd19227136aa3bb        cacheId
##  6: 556c3fbd14f8cdcf8bd19227136aa3bb     objectName
##  7: 556c3fbd14f8cdcf8bd19227136aa3bb       function
##  8: 556c3fbd14f8cdcf8bd19227136aa3bb    object.size
##  9: 556c3fbd14f8cdcf8bd19227136aa3bb       accessed
## 10: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 11: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 12: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 13: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 14: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 15: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 16: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 17: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 18: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 19: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 20: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 21: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 22: 556c3fbd14f8cdcf8bd19227136aa3bb      preDigest
## 23: 556c3fbd14f8cdcf8bd19227136aa3bb      preDigest
## 24: 7450fdaf3a485327278cec0e06f7472e         format
## 25: 7450fdaf3a485327278cec0e06f7472e           name
## 26: 7450fdaf3a485327278cec0e06f7472e          class
## 27: 7450fdaf3a485327278cec0e06f7472e           date
## 28: 7450fdaf3a485327278cec0e06f7472e        cacheId
## 29: 7450fdaf3a485327278cec0e06f7472e     objectName
## 30: 7450fdaf3a485327278cec0e06f7472e       function
## 31: 7450fdaf3a485327278cec0e06f7472e    object.size
## 32: 7450fdaf3a485327278cec0e06f7472e       accessed
## 33: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 34: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 35: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 36: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 37: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 38: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 39: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 40: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 41: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 42: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 43: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 44: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 45: 7450fdaf3a485327278cec0e06f7472e      preDigest
## 46: 7450fdaf3a485327278cec0e06f7472e      preDigest
##                             artifact         tagKey
##                                  tagValue         createdDate
##  1:                                   rda 2018-09-11 17:27:02
##  2:      556c3fbd14f8cdcf8bd19227136aa3bb 2018-09-11 17:27:02
##  3:                               numeric 2018-09-11 17:27:02
##  4:                   2018-09-11 17:27:02 2018-09-11 17:27:03
##  5:      f55d44974d91fbf61b6cb524983b1ebc 2018-09-11 17:27:03
##  6:                                     b 2018-09-11 17:27:03
##  7:                                 rnorm 2018-09-11 17:27:03
##  8:                                  1024 2018-09-11 17:27:03
##  9:                   2018-09-11 17:27:02 2018-09-11 17:27:03
## 10:                   withCallingHandlers 2018-09-11 17:27:03
## 11:                               saveRDS 2018-09-11 17:27:03
## 12:                               do.call 2018-09-11 17:27:03
## 13:                          process_file 2018-09-11 17:27:03
## 14:                         process_group 2018-09-11 17:27:03
## 15:                   process_group.block 2018-09-11 17:27:03
## 16:                            call_block 2018-09-11 17:27:03
## 17:                            block_exec 2018-09-11 17:27:03
## 18:                                in_dir 2018-09-11 17:27:03
## 19:                             timing_fn 2018-09-11 17:27:03
## 20:                                handle 2018-09-11 17:27:03
## 21:                           withVisible 2018-09-11 17:27:03
## 22:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:27:03
## 23: .FUN:a979d0700a6ef8c7c18736096ff1d522 2018-09-11 17:27:03
## 24:                                   rda 2018-09-11 17:27:02
## 25:      7450fdaf3a485327278cec0e06f7472e 2018-09-11 17:27:02
## 26:                               numeric 2018-09-11 17:27:02
## 27:                   2018-09-11 17:27:02 2018-09-11 17:27:02
## 28:      7cb09aa387ba235db013863648303b34 2018-09-11 17:27:02
## 29:                                     a 2018-09-11 17:27:02
## 30:                                 runif 2018-09-11 17:27:02
## 31:                                  1024 2018-09-11 17:27:02
## 32:                   2018-09-11 17:27:02 2018-09-11 17:27:02
## 33:                   withCallingHandlers 2018-09-11 17:27:02
## 34:                               saveRDS 2018-09-11 17:27:02
## 35:                               do.call 2018-09-11 17:27:02
## 36:                          process_file 2018-09-11 17:27:02
## 37:                         process_group 2018-09-11 17:27:02
## 38:                   process_group.block 2018-09-11 17:27:02
## 39:                            call_block 2018-09-11 17:27:02
## 40:                            block_exec 2018-09-11 17:27:02
## 41:                                in_dir 2018-09-11 17:27:02
## 42:                             timing_fn 2018-09-11 17:27:02
## 43:                                handle 2018-09-11 17:27:02
## 44:                           withVisible 2018-09-11 17:27:02
## 45:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:27:02
## 46: .FUN:179092fbad3ec261e58bc89bc8a703c3 2018-09-11 17:27:02
##                                  tagValue         createdDate
# keep only objects that are either runif or rnorm ("or" search)
keepCache(tmpDir, userTags = "runif|rnorm")
## Cache size:
##   Total (including Rasters): 512 bytes
##   Selected objects (not including Rasters): 512 bytes
##                             artifact         tagKey
##  1: 556c3fbd14f8cdcf8bd19227136aa3bb         format
##  2: 556c3fbd14f8cdcf8bd19227136aa3bb           name
##  3: 556c3fbd14f8cdcf8bd19227136aa3bb          class
##  4: 556c3fbd14f8cdcf8bd19227136aa3bb           date
##  5: 556c3fbd14f8cdcf8bd19227136aa3bb        cacheId
##  6: 556c3fbd14f8cdcf8bd19227136aa3bb     objectName
##  7: 556c3fbd14f8cdcf8bd19227136aa3bb       function
##  8: 556c3fbd14f8cdcf8bd19227136aa3bb    object.size
##  9: 556c3fbd14f8cdcf8bd19227136aa3bb       accessed
## 10: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 11: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 12: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 13: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 14: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 15: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 16: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 17: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 18: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 19: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 20: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 21: 556c3fbd14f8cdcf8bd19227136aa3bb otherFunctions
## 22: 556c3fbd14f8cdcf8bd19227136aa3bb      preDigest
## 23: 556c3fbd14f8cdcf8bd19227136aa3bb      preDigest
## 24: 7450fdaf3a485327278cec0e06f7472e         format
## 25: 7450fdaf3a485327278cec0e06f7472e           name
## 26: 7450fdaf3a485327278cec0e06f7472e          class
## 27: 7450fdaf3a485327278cec0e06f7472e           date
## 28: 7450fdaf3a485327278cec0e06f7472e        cacheId
## 29: 7450fdaf3a485327278cec0e06f7472e     objectName
## 30: 7450fdaf3a485327278cec0e06f7472e       function
## 31: 7450fdaf3a485327278cec0e06f7472e    object.size
## 32: 7450fdaf3a485327278cec0e06f7472e       accessed
## 33: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 34: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 35: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 36: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 37: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 38: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 39: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 40: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 41: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 42: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 43: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 44: 7450fdaf3a485327278cec0e06f7472e otherFunctions
## 45: 7450fdaf3a485327278cec0e06f7472e      preDigest
## 46: 7450fdaf3a485327278cec0e06f7472e      preDigest
##                             artifact         tagKey
##                                  tagValue         createdDate
##  1:                                   rda 2018-09-11 17:27:02
##  2:      556c3fbd14f8cdcf8bd19227136aa3bb 2018-09-11 17:27:02
##  3:                               numeric 2018-09-11 17:27:02
##  4:                   2018-09-11 17:27:02 2018-09-11 17:27:03
##  5:      f55d44974d91fbf61b6cb524983b1ebc 2018-09-11 17:27:03
##  6:                                     b 2018-09-11 17:27:03
##  7:                                 rnorm 2018-09-11 17:27:03
##  8:                                  1024 2018-09-11 17:27:03
##  9:                   2018-09-11 17:27:02 2018-09-11 17:27:03
## 10:                   withCallingHandlers 2018-09-11 17:27:03
## 11:                               saveRDS 2018-09-11 17:27:03
## 12:                               do.call 2018-09-11 17:27:03
## 13:                          process_file 2018-09-11 17:27:03
## 14:                         process_group 2018-09-11 17:27:03
## 15:                   process_group.block 2018-09-11 17:27:03
## 16:                            call_block 2018-09-11 17:27:03
## 17:                            block_exec 2018-09-11 17:27:03
## 18:                                in_dir 2018-09-11 17:27:03
## 19:                             timing_fn 2018-09-11 17:27:03
## 20:                                handle 2018-09-11 17:27:03
## 21:                           withVisible 2018-09-11 17:27:03
## 22:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:27:03
## 23: .FUN:a979d0700a6ef8c7c18736096ff1d522 2018-09-11 17:27:03
## 24:                                   rda 2018-09-11 17:27:02
## 25:      7450fdaf3a485327278cec0e06f7472e 2018-09-11 17:27:02
## 26:                               numeric 2018-09-11 17:27:02
## 27:                   2018-09-11 17:27:02 2018-09-11 17:27:02
## 28:      7cb09aa387ba235db013863648303b34 2018-09-11 17:27:02
## 29:                                     a 2018-09-11 17:27:02
## 30:                                 runif 2018-09-11 17:27:02
## 31:                                  1024 2018-09-11 17:27:02
## 32:                   2018-09-11 17:27:02 2018-09-11 17:27:02
## 33:                   withCallingHandlers 2018-09-11 17:27:02
## 34:                               saveRDS 2018-09-11 17:27:02
## 35:                               do.call 2018-09-11 17:27:02
## 36:                          process_file 2018-09-11 17:27:02
## 37:                         process_group 2018-09-11 17:27:02
## 38:                   process_group.block 2018-09-11 17:27:02
## 39:                            call_block 2018-09-11 17:27:02
## 40:                            block_exec 2018-09-11 17:27:02
## 41:                                in_dir 2018-09-11 17:27:02
## 42:                             timing_fn 2018-09-11 17:27:02
## 43:                                handle 2018-09-11 17:27:02
## 44:                           withVisible 2018-09-11 17:27:02
## 45:    n:c1de94a5a8d2a5813e07152fb7ea2038 2018-09-11 17:27:02
## 46: .FUN:179092fbad3ec261e58bc89bc8a703c3 2018-09-11 17:27:02
##                                  tagValue         createdDate
clearCache(tmpDir)

Example 5: using caching to speed up rerunning expensive computations

ras <- raster(extent(0, 5, 0, 5), res = 1,
              vals = sample(1:5, replace = TRUE, size = 25),
              crs = "+proj=lcc +lat_1=48 +lat_2=33 +lon_0=-100 +ellps=WGS84")

# A slow operation, like GIS operation
notCached <- suppressWarnings(
  # project raster generates warnings when run non-interactively
  projectRaster(ras, crs = crs(ras), res = 5, cacheRepo = tmpDir)
)

cached <- suppressWarnings(
  # project raster generates warnings when run non-interactively
  # using quote works also
  Cache(projectRaster, ras, crs = crs(ras), res = 5, cacheRepo = tmpDir)
)

# second time is much faster
reRun <- suppressWarnings(
  # project raster generates warnings when run non-interactively
  Cache(projectRaster, ras, crs = crs(ras), res = 5, cacheRepo = tmpDir)
)
##   loading cached result from previous projectRaster call, adding to memoised copy
# recovered cached version is same as non-cached version
all.equal(notCached, reRun) ## TRUE
## [1] TRUE

Nested Caching

Nested caching, which is when Caching of a function occurs inside an outer function, which is itself cached. This is a critical element to working within a reproducible work flow. It is not enough during development to cache flat code chunks, as there will be many levels of “slow” functions. Ideally, at all points in a development cycle, it should be possible to get to any line of code starting from the very initial steps, running through everything up to that point, in less that 1 second. If the workflow can be kept very fast like this, then there is a guarantee that it will work at any point.

##########################
## Nested Caching
# Make 2 functions
inner <- function(mean) {
  d <- 1
  Cache(rnorm, n = 3, mean = mean)
}
outer <- function(n) {
  Cache(inner, 0.1, cacheRepo = tmpdir2)
}

# make 2 different cache paths
tmpdir1 <- file.path(tempdir(), "first")
tmpdir2 <- file.path(tempdir(), "second")

# Run the Cache ... notOlderThan propagates to all 3 Cache calls,
#   but cacheRepo is tmpdir1 in top level Cache and all nested
#   Cache calls, unless individually overridden ... here inner
#   uses tmpdir2 repository
Cache(outer, n = 2, cacheRepo = tmpdir1, notOlderThan = Sys.time())
## [1] -0.41728961  0.02436647 -1.17721114
## attr(,"tags")
## [1] "cacheId:7360b42d0a2ddade58d480f1f9e9eef4"
## attr(,".Cache")
## attr(,".Cache")$newCache
## [1] TRUE
## 
## attr(,"call")
## [1] ""
showCache(tmpdir1) # 2 function calls
## Cache size:
##   Total (including Rasters): 512 bytes
##   Selected objects (not including Rasters): 512 bytes
##                             artifact         tagKey
##  1: 5acad470e17511faf1db6c4e60906ae3         format
##  2: 5acad470e17511faf1db6c4e60906ae3           name
##  3: 5acad470e17511faf1db6c4e60906ae3          class
##  4: 5acad470e17511faf1db6c4e60906ae3           date
##  5: 5acad470e17511faf1db6c4e60906ae3        cacheId
##  6: 5acad470e17511faf1db6c4e60906ae3       function
##  7: 5acad470e17511faf1db6c4e60906ae3    object.size
##  8: 5acad470e17511faf1db6c4e60906ae3       accessed
##  9: 5acad470e17511faf1db6c4e60906ae3 otherFunctions
## 10: 5acad470e17511faf1db6c4e60906ae3 otherFunctions
## 11: 5acad470e17511faf1db6c4e60906ae3 otherFunctions
## 12: 5acad470e17511faf1db6c4e60906ae3 otherFunctions
## 13: 5acad470e17511faf1db6c4e60906ae3 otherFunctions
## 14: 5acad470e17511faf1db6c4e60906ae3 otherFunctions
## 15: 5acad470e17511faf1db6c4e60906ae3 otherFunctions
## 16: 5acad470e17511faf1db6c4e60906ae3 otherFunctions
## 17: 5acad470e17511faf1db6c4e60906ae3 otherFunctions
## 18: 5acad470e17511faf1db6c4e60906ae3 otherFunctions
## 19: 5acad470e17511faf1db6c4e60906ae3 otherFunctions
## 20: 5acad470e17511faf1db6c4e60906ae3 otherFunctions
## 21: 5acad470e17511faf1db6c4e60906ae3      preDigest
## 22: 5acad470e17511faf1db6c4e60906ae3      preDigest
## 23: 5acad470e17511faf1db6c4e60906ae3      preDigest
## 24: 77b87849a4e6580edbd99b086b0e155f         format
## 25: 77b87849a4e6580edbd99b086b0e155f           name
## 26: 77b87849a4e6580edbd99b086b0e155f          class
## 27: 77b87849a4e6580edbd99b086b0e155f           date
## 28: 77b87849a4e6580edbd99b086b0e155f        cacheId
## 29: 77b87849a4e6580edbd99b086b0e155f       function
## 30: 77b87849a4e6580edbd99b086b0e155f    object.size
## 31: 77b87849a4e6580edbd99b086b0e155f       accessed
## 32: 77b87849a4e6580edbd99b086b0e155f otherFunctions
## 33: 77b87849a4e6580edbd99b086b0e155f otherFunctions
## 34: 77b87849a4e6580edbd99b086b0e155f otherFunctions
## 35: 77b87849a4e6580edbd99b086b0e155f otherFunctions
## 36: 77b87849a4e6580edbd99b086b0e155f otherFunctions
## 37: 77b87849a4e6580edbd99b086b0e155f otherFunctions
## 38: 77b87849a4e6580edbd99b086b0e155f otherFunctions
## 39: 77b87849a4e6580edbd99b086b0e155f otherFunctions
## 40: 77b87849a4e6580edbd99b086b0e155f otherFunctions
## 41: 77b87849a4e6580edbd99b086b0e155f otherFunctions
## 42: 77b87849a4e6580edbd99b086b0e155f otherFunctions
## 43: 77b87849a4e6580edbd99b086b0e155f otherFunctions
## 44: 77b87849a4e6580edbd99b086b0e155f      preDigest
## 45: 77b87849a4e6580edbd99b086b0e155f      preDigest
##                             artifact         tagKey
##                                  tagValue         createdDate
##  1:                                   rda 2018-09-11 17:27:04
##  2:      5acad470e17511faf1db6c4e60906ae3 2018-09-11 17:27:04
##  3:                               numeric 2018-09-11 17:27:04
##  4:                   2018-09-11 17:27:04 2018-09-11 17:27:04
##  5:      696a8c27b0af4d8124a69d16123c2a9b 2018-09-11 17:27:04
##  6:                                 rnorm 2018-09-11 17:27:04
##  7:                                  1024 2018-09-11 17:27:04
##  8:                   2018-09-11 17:27:04 2018-09-11 17:27:04
##  9:                   withCallingHandlers 2018-09-11 17:27:04
## 10:                               saveRDS 2018-09-11 17:27:04
## 11:                               do.call 2018-09-11 17:27:04
## 12:                          process_file 2018-09-11 17:27:04
## 13:                         process_group 2018-09-11 17:27:04
## 14:                   process_group.block 2018-09-11 17:27:04
## 15:                            call_block 2018-09-11 17:27:04
## 16:                            block_exec 2018-09-11 17:27:04
## 17:                                in_dir 2018-09-11 17:27:04
## 18:                             timing_fn 2018-09-11 17:27:04
## 19:                                handle 2018-09-11 17:27:04
## 20:                           withVisible 2018-09-11 17:27:04
## 21:    n:70822253d659facdcbfdb8a038cce013 2018-09-11 17:27:04
## 22: mean:e48ada84553a52f78ea2567bd2bc806a 2018-09-11 17:27:04
## 23: .FUN:a979d0700a6ef8c7c18736096ff1d522 2018-09-11 17:27:04
## 24:                                   rda 2018-09-11 17:27:04
## 25:      77b87849a4e6580edbd99b086b0e155f 2018-09-11 17:27:04
## 26:                               numeric 2018-09-11 17:27:04
## 27:                   2018-09-11 17:27:04 2018-09-11 17:27:04
## 28:      7360b42d0a2ddade58d480f1f9e9eef4 2018-09-11 17:27:04
## 29:                                 outer 2018-09-11 17:27:04
## 30:                                  1024 2018-09-11 17:27:04
## 31:                   2018-09-11 17:27:04 2018-09-11 17:27:04
## 32:                   withCallingHandlers 2018-09-11 17:27:04
## 33:                               saveRDS 2018-09-11 17:27:05
## 34:                               do.call 2018-09-11 17:27:05
## 35:                          process_file 2018-09-11 17:27:05
## 36:                         process_group 2018-09-11 17:27:05
## 37:                   process_group.block 2018-09-11 17:27:05
## 38:                            call_block 2018-09-11 17:27:05
## 39:                            block_exec 2018-09-11 17:27:05
## 40:                                in_dir 2018-09-11 17:27:05
## 41:                             timing_fn 2018-09-11 17:27:05
## 42:                                handle 2018-09-11 17:27:05
## 43:                           withVisible 2018-09-11 17:27:05
## 44:    n:bfc3454880d65ea9fc34069951dbbc35 2018-09-11 17:27:05
## 45: .FUN:dfb4fe3bd3b241021e1799f8234bd28f 2018-09-11 17:27:05
##                                  tagValue         createdDate
showCache(tmpdir2) # 1 function call
## Cache size:
##   Total (including Rasters): 256 bytes
##   Selected objects (not including Rasters): 256 bytes
##                             artifact         tagKey
##  1: 2c60846e3474f87e9315dd124c55ba6a         format
##  2: 2c60846e3474f87e9315dd124c55ba6a           name
##  3: 2c60846e3474f87e9315dd124c55ba6a          class
##  4: 2c60846e3474f87e9315dd124c55ba6a           date
##  5: 2c60846e3474f87e9315dd124c55ba6a        cacheId
##  6: 2c60846e3474f87e9315dd124c55ba6a       function
##  7: 2c60846e3474f87e9315dd124c55ba6a    object.size
##  8: 2c60846e3474f87e9315dd124c55ba6a       accessed
##  9: 2c60846e3474f87e9315dd124c55ba6a otherFunctions
## 10: 2c60846e3474f87e9315dd124c55ba6a otherFunctions
## 11: 2c60846e3474f87e9315dd124c55ba6a otherFunctions
## 12: 2c60846e3474f87e9315dd124c55ba6a otherFunctions
## 13: 2c60846e3474f87e9315dd124c55ba6a otherFunctions
## 14: 2c60846e3474f87e9315dd124c55ba6a otherFunctions
## 15: 2c60846e3474f87e9315dd124c55ba6a otherFunctions
## 16: 2c60846e3474f87e9315dd124c55ba6a otherFunctions
## 17: 2c60846e3474f87e9315dd124c55ba6a otherFunctions
## 18: 2c60846e3474f87e9315dd124c55ba6a otherFunctions
## 19: 2c60846e3474f87e9315dd124c55ba6a otherFunctions
## 20: 2c60846e3474f87e9315dd124c55ba6a otherFunctions
## 21: 2c60846e3474f87e9315dd124c55ba6a      preDigest
## 22: 2c60846e3474f87e9315dd124c55ba6a      preDigest
##                             artifact         tagKey
##                                  tagValue         createdDate
##  1:                                   rda 2018-09-11 17:27:04
##  2:      2c60846e3474f87e9315dd124c55ba6a 2018-09-11 17:27:04
##  3:                               numeric 2018-09-11 17:27:04
##  4:                   2018-09-11 17:27:04 2018-09-11 17:27:04
##  5:      3d9566c2b0275c7478ee84c56177b51c 2018-09-11 17:27:04
##  6:                                 inner 2018-09-11 17:27:04
##  7:                                  1024 2018-09-11 17:27:04
##  8:                   2018-09-11 17:27:04 2018-09-11 17:27:04
##  9:                   withCallingHandlers 2018-09-11 17:27:04
## 10:                               saveRDS 2018-09-11 17:27:04
## 11:                               do.call 2018-09-11 17:27:04
## 12:                          process_file 2018-09-11 17:27:04
## 13:                         process_group 2018-09-11 17:27:04
## 14:                   process_group.block 2018-09-11 17:27:04
## 15:                            call_block 2018-09-11 17:27:04
## 16:                            block_exec 2018-09-11 17:27:04
## 17:                                in_dir 2018-09-11 17:27:04
## 18:                             timing_fn 2018-09-11 17:27:04
## 19:                                handle 2018-09-11 17:27:04
## 20:                           withVisible 2018-09-11 17:27:04
## 21: mean:e48ada84553a52f78ea2567bd2bc806a 2018-09-11 17:27:04
## 22: .FUN:cf56d34c48d5d8c8a0ac5bd78262cdd9 2018-09-11 17:27:04
##                                  tagValue         createdDate
# userTags get appended
# all items have the outer tag propagate, plus inner ones only have inner ones
clearCache(tmpdir1)
outerTag <- "outerTag"
innerTag <- "innerTag"
inner <- function(mean) {
  d <- 1
  Cache(rnorm, n = 3, mean = mean, notOlderThan = Sys.time() - 1e5, userTags = innerTag)
}
outer <- function(n) {
  Cache(inner, 0.1)
}
aa <- Cache(outer, n = 2, cacheRepo = tmpdir1, userTags = outerTag)
showCache(tmpdir1) # rnorm function has outerTag and innerTag, inner and outer only have outerTag
## Cache size:
##   Total (including Rasters): 768 bytes
##   Selected objects (not including Rasters): 768 bytes
##                             artifact         tagKey
##  1: 48560a253b62616964c18ef733f29c8a         format
##  2: 48560a253b62616964c18ef733f29c8a           name
##  3: 48560a253b62616964c18ef733f29c8a          class
##  4: 48560a253b62616964c18ef733f29c8a           date
##  5: 48560a253b62616964c18ef733f29c8a        cacheId
##  6: 48560a253b62616964c18ef733f29c8a       innerTag
##  7: 48560a253b62616964c18ef733f29c8a       outerTag
##  8: 48560a253b62616964c18ef733f29c8a       function
##  9: 48560a253b62616964c18ef733f29c8a    object.size
## 10: 48560a253b62616964c18ef733f29c8a       accessed
## 11: 48560a253b62616964c18ef733f29c8a otherFunctions
## 12: 48560a253b62616964c18ef733f29c8a otherFunctions
## 13: 48560a253b62616964c18ef733f29c8a otherFunctions
## 14: 48560a253b62616964c18ef733f29c8a otherFunctions
## 15: 48560a253b62616964c18ef733f29c8a otherFunctions
## 16: 48560a253b62616964c18ef733f29c8a otherFunctions
## 17: 48560a253b62616964c18ef733f29c8a otherFunctions
## 18: 48560a253b62616964c18ef733f29c8a otherFunctions
## 19: 48560a253b62616964c18ef733f29c8a otherFunctions
## 20: 48560a253b62616964c18ef733f29c8a otherFunctions
## 21: 48560a253b62616964c18ef733f29c8a otherFunctions
## 22: 48560a253b62616964c18ef733f29c8a otherFunctions
## 23: 48560a253b62616964c18ef733f29c8a      preDigest
## 24: 48560a253b62616964c18ef733f29c8a      preDigest
## 25: 48560a253b62616964c18ef733f29c8a      preDigest
## 26: 67a0ede28e9338c61181149b9d5840d9         format
## 27: 67a0ede28e9338c61181149b9d5840d9           name
## 28: 67a0ede28e9338c61181149b9d5840d9          class
## 29: 67a0ede28e9338c61181149b9d5840d9           date
## 30: 67a0ede28e9338c61181149b9d5840d9        cacheId
## 31: 67a0ede28e9338c61181149b9d5840d9       outerTag
## 32: 67a0ede28e9338c61181149b9d5840d9       function
## 33: 67a0ede28e9338c61181149b9d5840d9    object.size
## 34: 67a0ede28e9338c61181149b9d5840d9       accessed
## 35: 67a0ede28e9338c61181149b9d5840d9 otherFunctions
## 36: 67a0ede28e9338c61181149b9d5840d9 otherFunctions
## 37: 67a0ede28e9338c61181149b9d5840d9 otherFunctions
## 38: 67a0ede28e9338c61181149b9d5840d9 otherFunctions
## 39: 67a0ede28e9338c61181149b9d5840d9 otherFunctions
## 40: 67a0ede28e9338c61181149b9d5840d9 otherFunctions
## 41: 67a0ede28e9338c61181149b9d5840d9 otherFunctions
## 42: 67a0ede28e9338c61181149b9d5840d9 otherFunctions
## 43: 67a0ede28e9338c61181149b9d5840d9 otherFunctions
## 44: 67a0ede28e9338c61181149b9d5840d9 otherFunctions
## 45: 67a0ede28e9338c61181149b9d5840d9 otherFunctions
## 46: 67a0ede28e9338c61181149b9d5840d9 otherFunctions
## 47: 67a0ede28e9338c61181149b9d5840d9      preDigest
## 48: 67a0ede28e9338c61181149b9d5840d9      preDigest
## 49: fe71b24676cc4a9999fb598ef6a2801a         format
## 50: fe71b24676cc4a9999fb598ef6a2801a           name
## 51: fe71b24676cc4a9999fb598ef6a2801a          class
## 52: fe71b24676cc4a9999fb598ef6a2801a           date
## 53: fe71b24676cc4a9999fb598ef6a2801a        cacheId
## 54: fe71b24676cc4a9999fb598ef6a2801a       outerTag
## 55: fe71b24676cc4a9999fb598ef6a2801a       function
## 56: fe71b24676cc4a9999fb598ef6a2801a    object.size
## 57: fe71b24676cc4a9999fb598ef6a2801a       accessed
## 58: fe71b24676cc4a9999fb598ef6a2801a otherFunctions
## 59: fe71b24676cc4a9999fb598ef6a2801a otherFunctions
## 60: fe71b24676cc4a9999fb598ef6a2801a otherFunctions
## 61: fe71b24676cc4a9999fb598ef6a2801a otherFunctions
## 62: fe71b24676cc4a9999fb598ef6a2801a otherFunctions
## 63: fe71b24676cc4a9999fb598ef6a2801a otherFunctions
## 64: fe71b24676cc4a9999fb598ef6a2801a otherFunctions
## 65: fe71b24676cc4a9999fb598ef6a2801a otherFunctions
## 66: fe71b24676cc4a9999fb598ef6a2801a otherFunctions
## 67: fe71b24676cc4a9999fb598ef6a2801a otherFunctions
## 68: fe71b24676cc4a9999fb598ef6a2801a otherFunctions
## 69: fe71b24676cc4a9999fb598ef6a2801a otherFunctions
## 70: fe71b24676cc4a9999fb598ef6a2801a      preDigest
## 71: fe71b24676cc4a9999fb598ef6a2801a      preDigest
##                             artifact         tagKey
##                                  tagValue         createdDate
##  1:                                   rda 2018-09-11 17:27:05
##  2:      48560a253b62616964c18ef733f29c8a 2018-09-11 17:27:05
##  3:                               numeric 2018-09-11 17:27:05
##  4:                   2018-09-11 17:27:05 2018-09-11 17:27:05
##  5:      696a8c27b0af4d8124a69d16123c2a9b 2018-09-11 17:27:05
##  6:                              innerTag 2018-09-11 17:27:05
##  7:                              outerTag 2018-09-11 17:27:05
##  8:                                 rnorm 2018-09-11 17:27:05
##  9:                                  1024 2018-09-11 17:27:05
## 10:                   2018-09-11 17:27:05 2018-09-11 17:27:05
## 11:                   withCallingHandlers 2018-09-11 17:27:05
## 12:                               saveRDS 2018-09-11 17:27:05
## 13:                               do.call 2018-09-11 17:27:05
## 14:                          process_file 2018-09-11 17:27:05
## 15:                         process_group 2018-09-11 17:27:05
## 16:                   process_group.block 2018-09-11 17:27:05
## 17:                            call_block 2018-09-11 17:27:05
## 18:                            block_exec 2018-09-11 17:27:05
## 19:                                in_dir 2018-09-11 17:27:05
## 20:                             timing_fn 2018-09-11 17:27:05
## 21:                                handle 2018-09-11 17:27:05
## 22:                           withVisible 2018-09-11 17:27:05
## 23:    n:70822253d659facdcbfdb8a038cce013 2018-09-11 17:27:05
## 24: mean:e48ada84553a52f78ea2567bd2bc806a 2018-09-11 17:27:05
## 25: .FUN:a979d0700a6ef8c7c18736096ff1d522 2018-09-11 17:27:05
## 26:                                   rda 2018-09-11 17:27:05
## 27:      67a0ede28e9338c61181149b9d5840d9 2018-09-11 17:27:05
## 28:                               numeric 2018-09-11 17:27:05
## 29:                   2018-09-11 17:27:05 2018-09-11 17:27:05
## 30:      21818a9ce14c5526152af5f6bd649f1c 2018-09-11 17:27:05
## 31:                              outerTag 2018-09-11 17:27:05
## 32:                                 inner 2018-09-11 17:27:05
## 33:                                  1024 2018-09-11 17:27:05
## 34:                   2018-09-11 17:27:05 2018-09-11 17:27:05
## 35:                   withCallingHandlers 2018-09-11 17:27:05
## 36:                               saveRDS 2018-09-11 17:27:05
## 37:                               do.call 2018-09-11 17:27:05
## 38:                          process_file 2018-09-11 17:27:05
## 39:                         process_group 2018-09-11 17:27:05
## 40:                   process_group.block 2018-09-11 17:27:05
## 41:                            call_block 2018-09-11 17:27:05
## 42:                            block_exec 2018-09-11 17:27:05
## 43:                                in_dir 2018-09-11 17:27:05
## 44:                             timing_fn 2018-09-11 17:27:05
## 45:                                handle 2018-09-11 17:27:05
## 46:                           withVisible 2018-09-11 17:27:05
## 47: mean:e48ada84553a52f78ea2567bd2bc806a 2018-09-11 17:27:05
## 48: .FUN:7371e13cb8e6f554d8e048c938825ead 2018-09-11 17:27:05
## 49:                                   rda 2018-09-11 17:27:06
## 50:      fe71b24676cc4a9999fb598ef6a2801a 2018-09-11 17:27:06
## 51:                               numeric 2018-09-11 17:27:06
## 52:                   2018-09-11 17:27:06 2018-09-11 17:27:06
## 53:      527dd0f4adb50e543f78626baca662a7 2018-09-11 17:27:06
## 54:                              outerTag 2018-09-11 17:27:06
## 55:                                 outer 2018-09-11 17:27:06
## 56:                                  1024 2018-09-11 17:27:06
## 57:                   2018-09-11 17:27:05 2018-09-11 17:27:06
## 58:                   withCallingHandlers 2018-09-11 17:27:06
## 59:                               saveRDS 2018-09-11 17:27:06
## 60:                               do.call 2018-09-11 17:27:06
## 61:                          process_file 2018-09-11 17:27:06
## 62:                         process_group 2018-09-11 17:27:06
## 63:                   process_group.block 2018-09-11 17:27:06
## 64:                            call_block 2018-09-11 17:27:06
## 65:                            block_exec 2018-09-11 17:27:06
## 66:                                in_dir 2018-09-11 17:27:06
## 67:                             timing_fn 2018-09-11 17:27:06
## 68:                                handle 2018-09-11 17:27:06
## 69:                           withVisible 2018-09-11 17:27:06
## 70:    n:bfc3454880d65ea9fc34069951dbbc35 2018-09-11 17:27:06
## 71: .FUN:ccd653e1f8453c8e3dcd49c99f824ed9 2018-09-11 17:27:06
##                                  tagValue         createdDate

cacheId

Sometimes, it is not absolutely desirable to maintain the work flow intact because changes that are irrelevant to the analysis, such as changing messages sent to a user, may be changed, without a desire to rerun functions. The cacheId argument is for this. Once a piece of code is run, then the cacheId can be manually extracted (it is reported at the end of a Cache call) and manually placed in the code, passed in as, say, cacheId = "ad184ce64541972b50afd8e7b75f821b".

### cacheId
set.seed(1)
Cache(rnorm, 1, cacheRepo = tmpdir1)
## [1] -0.6264538
## attr(,"tags")
## [1] "cacheId:a5fee427faceb81b71e68a2f834d377b"
## attr(,".Cache")
## attr(,".Cache")$newCache
## [1] TRUE
## 
## attr(,"call")
## [1] ""
# manually look at output attribute which shows cacheId: ad184ce64541972b50afd8e7b75f821b
Cache(rnorm, 1, cacheRepo = tmpdir1, cacheId = "ad184ce64541972b50afd8e7b75f821b") # same value
## cacheId is not same as calculated hash. Manually searching for cacheId:ad184ce64541972b50afd8e7b75f821b
## [1] 0.1836433
## attr(,"tags")
## [1] "cacheId:ad184ce64541972b50afd8e7b75f821b"
## attr(,".Cache")
## attr(,".Cache")$newCache
## [1] TRUE
## 
## attr(,"call")
## [1] ""
# override even with different inputs:
Cache(rnorm, 2, cacheRepo = tmpdir1, cacheId = "ad184ce64541972b50afd8e7b75f821b")
## cacheId is not same as calculated hash. Manually searching for cacheId:ad184ce64541972b50afd8e7b75f821b
##   loading cached result from previous rnorm call, adding to memoised copy
## [1] 0.1836433
## attr(,"tags")
## [1] "cacheId:ad184ce64541972b50afd8e7b75f821b"
## attr(,".Cache")
## attr(,".Cache")$newCache
## [1] FALSE
## 
## attr(,"call")
## [1] ""
## cleanup
unlink(c("filename.rda", "filename1.rda"))

Working with the Cache manually

Since the cache is simply an archivist repository, all archivist functions will work as is. In addition, there are several helpers in the reproducible package, including showCache, keepCache and clearCache that may be useful. Also, one can access cached items manually (rather than simply rerunning the same Cache function again).

if (requireNamespace("archivist")) {
  # get the RasterLayer that was produced with the gaussMap function:
  mapHash <- unique(showCache(tmpDir, userTags = "projectRaster")$artifact)
  map <- archivist::loadFromLocalRepo(md5hash = mapHash[1], repoDir = tmpDir, value = TRUE)
  
  plot(map)
}
## Cache size:
##   Total (including Rasters): 3.3 Kb
##   Selected objects (not including Rasters): 3.3 Kb

## cleanup
unlink(dirname(tmpDir), recursive = TRUE)

Reproducible Workflow

In general, we feel that a liberal use of Cache will make a re-usable and reproducible work flow. shiny apps can be made, taking advantage of Cache. Indeed, much of the difficulty in managing data sets and saving them for future use, can be accommodated by caching.

Nested Caching

Cache individual functions

Cache(<functionName>, <other arguments>)

This will allow fine scale control of individual function calls.