Alternative to prepInputs that can use Spatial Tiles stored locally or on Google Drive

Source: R/downloadTileAndUpload.R

Downloads, processes and optionally uploads a SpatRaster object through a tiling intermediary. If the original url is for a very large object, but to is a relatively small subset of the area represented by the spatial file at url, then this function will potentially by-pass the download of the large file at url and instead only download the minimum number of tiles necessary to cover the to area. When doUploads is TRUE, then this function will potentially create and upload the tiles to tileFolder, prior to returning the spatial object, postProcessed to to. This function supports both Google Drive and HTTP(S) URLs.

prepInputsWithTiles(
  targetFile,
  url,
  destinationPath,
  to,
  tilesFolder = file.path(getOption("reproducible.inputPath"), "tiles"),
  urlTiles = getOption("reproducible.prepInputsUrlTiles", NULL),
  doUploads = getOption("reproducible.prepInputsDoUploads", FALSE),
  tileGrid = "CAN",
  numTiles = NULL,
  plot.grid = FALSE,
  purge = FALSE,
  verbose = getOption("reproducible.verbose"),
  ...
)

Arguments

targetFile

Character. Name of the target file to be downloaded or processed. If missing, it will be inferred from the URL or Google Drive metadata.

url

Character. URL to the full dataset (Google Drive or HTTP/S).

destinationPath

Character. Path to the directory where files will be downloaded and processed.

to

A spatial object (e.g., SpatRaster, SpatVector, sf, or Spatial*) defining the area of interest.

tilesFolder

A local file path to put tiles. If this is an absolute path, then that will be used; if it is a relative path, then it will be file.path(destinationPath, tilesFolder)

urlTiles

Character. URL to the tile source (e.g., Google Drive folder or HTTP/S endpoint). Default is getOption("reproducible.prepInputsUrlTiles", NULL).

doUploads

Logical. Whether to upload processed tiles. Default is getOption("reproducible.prepInputsDoUploads", FALSE).

tileGrid

Either length 3 character string, such as "CAN", to be sent to geodata::gadm(...) or an actual SpatVector object with a grid of polygons

numTiles

Integer. Number of tiles to generate. Optional.

plot.grid

Logical. Whether to plot the tile grid and area of interest. Default is FALSE.

purge

Logical or Integer. 0/FALSE (default) keeps existing CHECKSUMS.txt file and prepInputs will write or append to it. 1/TRUE will deleted the entire CHECKSUMS.txt file.

verbose

Logical or numeric. Controls verbosity of messages. Default is getOption("reproducible.verbose").

...

Either maskTo, cropTo (which will be used if to is not supplied, or arguments passed to writeRaster, e.g., datatype (used when writing tiles).

Value

A single, merged SpatRaster object postProcessed to the area of interest (to), composed of the necessary tiles. If the post-processed file already exists locally, it will be returned directly.

Details

This function can be triggered inside prepInputs if the to is supplied and both url and urlTiles are supplied. NOTE: urlTiles can be supplied using the option(reproducible.prepInputsUrlTiles = someGoogleDriveFolderURL), so the original prepInputs function call can remain unaffected.

This function also uses a different checksumming procedure compared to the normal prepInputs. This function will assess the remote url for a hash. If that hash exists, then it will compare it to a local file with targetFile name, suffixed with .hash. If the two hashes differ (remote and local), then it will be redownloaded; otherwise the local one will be returned.

This function is useful for working with large spatial datasets, but where the user only requires a "relatively small" section of that dataset. This function will potentially bypass the full download and download only the tiles that are necessary for the to. It handles downloading only the required tiles based on spatial intersection with the target area, and supports resumable downloads from Google Drive or HTTP/S sources.

If targetFile is missing, the function attempts to infer it from the URL using the Content-Disposition header or the basename of the URL. For Google Drive URLs, it uses the file metadata.

Examples


if (FALSE) {
  to <- sf::st_as_sf(sf::st_sfc(sf::st_point(c(-123.3656, 48.4284)), crs = 4326))
  result <- prepInputsWithTiles(
    url = "https://example.com/data.tif",
    destinationPath = tempdir(),
    to = to,
    urlTiles = "https://example.com/tiles/",
    tileGrid = "CAN"
  )
}