Skip to contents

Assemble observations

Source: R/assemble-data.R
assemble.Rd

These functions assemble a timeline and observations for the particle filter (pf_filter()).

Usage

assemble_timeline(.datasets = list(), .step, .trim = FALSE)

assemble_acoustics(.timeline, .detections, .moorings, .services = NULL)

assemble_archival(.timeline, .archival)

assemble_custom(.timeline, .dataset)

assemble_xinit_containers(
  .timeline,
  .xinit = list(),
  .radius,
  .mobility,
  .map = NULL,
  .threshold = NULL
)

assemble_acoustics_containers(
  .timeline,
  .acoustics,
  .mobility,
  .map = NULL,
  .threshold = NULL
)

assemble_containers(...)

Arguments

.datasets, .step, .trim

Arguments for assemble_timeline().

  • .datasets—A list of data.table::data.tables, one for each data type, each containing a timestamp column;

  • .step—A character (such as "2 mins"), passed to lubridate::round_date() and seq.POSIXt(), that defines the resolution of the timeline;

  • .trim—A logical variable that defines whether or not to trim the timeline to the overlapping period between datasets;

.timeline

A POSIXct vector of regularly spaced time stamps that defines the timeline for the simulation (optionally from assemble_timeline()). Here, timeline is used to:

  • Define the resolution of observations;

.detections, .moorings, .services

The data.table::data.tables for assemble_acoustics() (see pat_setup_data()).

  • .detections is a data.table::data.table of acoustic detections for a single individual. This must contain the receiver_id and timestamp columns.

  • .moorings is a data.table::data.table of acoustic receiver deployments. This must contain the receiver_id, receiver_start, and receiver_end columns, plus (optional) additional parameter columns.

  • (optional) .services is a data.table::data.table of servicing events. This must contain the receiver_id, service_start and service_end columns.

.archival

For assemble_archival(), .archival is a data.table::data.table of depth observations for a single individual with timestamp and obs columns (see .dataset, below).

.dataset

For assemble_custom(), .dataset is a data.table::data.table of observations (such as depth measurements) for a single individual. This must contain timestamp and obs columns plus (optional) additional parameter columns.

.xinit, .radius,

Dataset arguments for assemble_xinit_containers().

  • .xinit is a named list, with elements "forward" and "backward". Elements should be NULL or a data.table::data.table of initial state(s) (i.e., capture or recapture locations) for the corresponding filter run. If capture/recapture locations are known exactly, use a one-row data.table::data.table. If starting/ending locations are not known exactly, multi-row data.table::data.tables (with all possible starting/ending locations) are permitted.

  • .radius is a double that defines the radius of the container around the capture/recapture location that particles must reach (in assemble_acoustics_containers(), .radius = .acoustics$detection_gamma).

.mobility, .map, .threshold

Shared container threshold arguments (for assemble_xinit_containers() and assemble_acoustics_containers()).

  • .mobility is the maximum movement distance (m) between two time steps (and sets the rate of container contraction).

  • .map, .threshold are distance threshold options. Specify .map or .threshold:

    • .map is a two-column matrix of the four coordinates of the study area or a terra::SpatRaster or terra::SpatVector from which such a matrix can be obtained. On Linux, the latter two options are only possible if JUILA_SESSION = "FALSE". .threshold is set automatically based on the distances between container centroids and the boundaries of the study area.

    • Otherwise, .threshold is a double that defines the distance threshold.

.acoustics

Dataset arguments for assemble_acoustics_containers().

...

For assemble_containers(), ... represents container lists for multiple data types, such as capture events (assemble_xinit_containers()) and acoustic observations (assemble_acoustics_containers()).

Value

  • assemble_timeline() returns a POSIXct vector;

  • assemble_acoustics(), assemble_archival() and assemble_custom() return a data.table::data.table for pf_filter();

  • assemble_xinit_containers(), assemble_acoustics_containers() and assemble_containers() return a named list, with one element (data.table::data.table) for (a) the forward and (b) the backward runs of pf_filter();

Assemble timeline

assemble_timeline() is a simple function that defines a regular timeline, of resolution .step, from a list of input datasets.

  • If .trim = FALSE, this defines a sequence of regular time stamps across the full range of time stamps in the input datasets.

  • If .trim = TRUE, the timeline is trimmed to the overlapping period between datasets.

Assemble datasets

assemble_{dataset}() functions are helper routines that prepare timelines observations for different data types as required for the particle filter (pf_filter()). The filter expects a named list of datasets (one for each data type). Each dataset must contain the following columns:

  • timestamp

  • sensor_id

  • obs

  • Additional columns with the parameters of the observation model (see glossary).

assemble_acoustics(), assemble_archival() and assemble_custom() assemble 'standard' observational time series:

  1. assemble_acoustics() prepares a timeline of acoustic observations as required by the filter for a single individual. This function expects a 'standard' detection dataset (that is, a data.table::data.table like dat_detections but for a single individual) that defines detections at receivers alongside a moorings dataset (like dat_moorings) that defines receiver deployment periods and, optionally, a data.table::data.table of servicing events (when receiver(s) were non-operational). assemble_acoustics() uses these datasets to assemble a complete time series of acoustic observations; that is, a data.table::data.table of time stamps and receivers that defines, for each time step and each operational receiver whether (1L) or not (0L) a detection was recorded at that time step. Duplicate observations (that is, detections at the same receiver in the same time step) are dropped. If available in .moorings, additional columns (receiver_alpha, receiver_beta and receiver_gamma) are included as required for the default acoustic observation model (that is, ModelObsAcousticLogisTrunc). If observation model parameters vary both by receiver and through time, simply amend these columns as required.

  2. assemble_archival() prepares a timeline of archival functions for a single individual. This simply wraps assemble_custom() and is informally deprecated.

  3. assemble_custom() prepares a timeline of observations for other data types, as required by the filter. This function expects a data.table::data.table that includes, at a minimum, the timestamp and obs columns. The latter defines the observations. The sensor_id column (if unspecified) is simply set to 1L. The function re-expresses time stamps at the resolution specified by timeline. Duplicate observations (that is, multiple measurements in the same time step) throw a warning.

Assemble containers

assemble_containers(), assemble_xinit_containers() and assemble_acoustics_containers() assemble 'container' datasets. A container is a circular region within which an individual must be located, given:

  • A 'future' observation in a particular place, such as a capture/recapture event or an acoustic detection;

  • The time until that observation;

  • An individual's maximum movement speed (.mobility);

As we approach a 'future' observation, the container within which an individual must be located, according to that observation, shrinks towards the location in which the observation was recorded. Encoding this knowledge in the particle filter (by killing particles outside of a container that are incompatible with a future observation) facilitates convergence with fewer particles, assuming regular resampling (see pf_filter()). (Otherwise, we rely on some particles ending up in the right region by chance, which can be unlikely if the region in which particles must end up is small (e.g., a receiver's detection container).)

Consider acoustic containers as an example. Suppose an individual can move up to .mobility = 500 m per time step, and two time steps elapse between the first and second detections, then at the moment of the first detection the individual must be within 1000 m of the detection range (say, receiver_gamma = 750 m) of the second receiver; that is, at the moment of first detection, the maximum possible distance of the individual from the receiver that recorded the next detection is 1750 m. As time passes, the container shrinks towards the receiver(s) that recorded the next detection(s), in line with the individual's .mobility. Pro-actively killing particles that move outside these containers (and are incompatible with the next detection) facilitates convergence.

The following functions assemble containers:

  1. assemble_xinit_containers() assembles containers for initial capture or recapture events. This is designed for situations in which you know the starting and/or ending location for an individual. Suppose you know the starting location of an individual. We can specify this in the forward filter run via .xinit in pf_filter(). But what about the backward filter run? For particle smoothing (via pf_smoother_two_filter()), we need a forward and a backward filter run and these must align sufficiently. Containers provide a mechanism that encourages the backward filter run to end up in the known starting location. Similarly, if we know a recapture location, containers in the forward filter run provide a mechanism that encourages the filter to end up in that location. In assemble_xinit_containers(), .xinit defines the initial locations for the forward/backward filter runs and .radius defines the radius of the starting/ending container. If you know capture/recapture locations exactly, strictly speaking .radius = 0. But it may help to permit some flexibility (e.g., .radius = .mobility) to facilitate convergence (it is unlikely that particles will finish in exactly the right location). If capture/recapture locations are not known exactly, the centroid of the input coordinates is taken as the container's centroid. The container radius at the time of capture/recapture event is given by the maximum distance from the centroid to any of the coordinates plus .radius. As time passes, containers shrink towards this region in line with .mobility.

  2. assemble_acoustics_containers() prepares a dataset of acoustic containers, given the acoustic time series from assemble_acoustics(). Acoustic containers define the region within which an individual must be located at a given time step according to the receiver(s) at which it was next detected. The radius depends on the time until the next detection, the maximum movement speed and the detection range around the receiver at the time of the detection. This function requires the tidyr::nest(), tidyr::unnest() and zoo::na.locf() functions (suggested dependencies).

  3. assemble_containers() is a post-processing function. Use this function to collate the container data.table::data.tables from multiple datasets, i.e., if you have capture/recapture and acoustic containers.

All assemble_*_containers() functions assemble a list of data.table::data.tables (with one element for the forward filter run and one element for the backward filter run). Each row defines the maximum distance (radius) of the individual from the location in which a future observation was recorded. For computational efficiency, data.table::data.tables only include containers with a radius < .threshold. If .map is supplied, the .threshold is set to the maximum distance between each location (e.g., receiver) and the furthest corner of the study area. Otherwise, set the .threshold to the desired value. The data.table::data.tables are used to instantiate a Vector of ModelObsContainer instances in Julia. (Only one sub-type is used for all container structures for speed.) In the particle filter (pf_filter()), for each particle, we compute the log-probability of the particle from the distance of the particle from the relevant location (0.0 or -Inf). By re-sampling particles with replacement, particles that move in a way that is incompatible with the future data (e.g., a detection) are killed.

Julia implementation

In Julia, datasets are translated into a hash-table (Dict) of observations (via Patter.assemble_yobs()). For each time stamp with an observation, this includes a Vector of Tuples, each containing the observation and the associated ModelObs instance that defines the parameters of the observation model. The particle filter (Patter.particle_filter()) iterates over each time step in the timeline, uses a movement model to simulate animal movement and, for the time stamps with observations, evaluates the likelihood of those observations for the simulated locations (particles).

assemble_*() routines are only required for real-world analyses.

See also

Particle filters and smoothers sample states (particles) that represent the possible locations of an individual through time, accounting for all data and the individual's movement.

Author

Edward Lavender

Examples

if (patter_run(.julia = FALSE, .geospatial = TRUE)) {

  library(data.table)
  library(dtplyr)
  library(dplyr, warn.conflicts = FALSE)

  #### Define example dataset(s) for a selected individual
  # Study area map
  map <- dat_gebco()
  # Acoustic detection time series
  # * Observation model parameters are defined in `.moorings`
  det <-
    dat_detections |>
    filter(individual_id == 25L) |>
    select("timestamp", "receiver_id") |>
    as.data.table()
  # Archival time series
  # * Observation model parameters must be included
  # * Here, we define parameters for `?ModelObsDepthNormalTruncSeabed`
  arc <-
    dat_archival |>
    filter(individual_id == 25L) |>
    select("timestamp", obs = "depth") |>
    mutate(depth_sigma = 50, depth_deep_eps = 20) |>
    as.data.table()

  #### Example (1): Define a timeline
  # Define a timeline manually
  timeline <- seq(as.POSIXct("2016-03-01 00:00:00", tz = "UTC"),
                  as.POSIXct("2016-04-01 00:00:00"),
                  by = "2 mins")
  # Use `assemble_timeline()` with `.trim = FALSE`
  timeline <- assemble_timeline(list(det, arc), .step = "2 mins")
  range(timeline)
  # Use `assemble_timeline()` with `.trim = TRUE`
  timeline <- assemble_timeline(list(det, arc), .step = "2 mins", .trim = TRUE)
  timeline <- timeline[1:1440]
  range(timeline)

  #### Example (2): Assemble an acoustic timeline
  # Assemble a timeline of acoustic observations (0, 1) and model parameters
  # * The default acoustic observation model parameters are taken from `.moorings`
  # * But can be modified or added afterwards for custom observation models
  acoustics <- assemble_acoustics(.timeline = timeline,
                                  .detections = det,
                                  .moorings = dat_moorings)
  head(acoustics)

  #### Example (3): Assemble an archival timeline
  # Assemble a timeline of archival observations and model parameters
  archival <- assemble_archival(.timeline = timeline,
                                .archival = arc)
  head(archival)

  #### Example (4): Assemble custom datasets
  temperature <-
    data.table(timestamp = c(as.POSIXct("2016-03-17 01:50:30", tz = "UTC"),
                             as.POSIXct("2016-03-17 02:00:30 UTC", tz = "UTC")),
               obs = c(7.6, 7.7))
  temperature <- assemble_custom(.timeline = timeline,
                                 .dataset = temperature)
  head(temperature)

  #### Example (5): Assemble xinit (capture/recapture) containers
  # (A) If we know the capture location, containers are defined for the backward filter run
  capture_xy       <- data.table(x = 708913.6, y = 6256280)
  xinit_containers <-
    assemble_xinit_containers(.timeline = timeline,
                              .xinit = list(forward = capture_xy,
                                            backward = NULL),
                              .radius = 750,
                              .mobility = 750,
                              .map = map)
  # (B) If we know the recapture location, containers are defined for the forward run
  recapture_xy     <- data.table(x = 707816.5, y = 6265746)
  xinit_containers <-
    assemble_xinit_containers(.timeline = timeline,
                              .xinit = list(forward = NULL,
                                            backward = recapture_xy),
                              .radius = 750,
                              .mobility = 750,
                              .map = map)
  # (C) If we know capture & recapture locations, containers are defined for both filter runs
  xinit_containers <-
    assemble_xinit_containers(.timeline = timeline,
                              .xinit = list(forward = capture_xy,
                                            backward = recapture_xy),
                              .radius = 750,
                              .mobility = 750,
                              .map = map)
  # (D) A set of possible capture/recapture locations is permitted:
  # (i) Define possible starting locations
  capture_xy <-
    cbind(capture_xy$x, capture_xy$y) |>
    terra::vect(crs = terra::crs(map)) |>
    terra::buffer(width = 500) |>
    terra::spatSample(size = 10L) |>
    terra::crds(df = TRUE) |>
    setDT()
  # (ii) Define possible recapture locations
  recapture_xy <-
    cbind(recapture_xy$x, recapture_xy$y) |>
    terra::vect(crs = terra::crs(map)) |>
    terra::buffer(width = 500) |>
    terra::spatSample(size = 10L) |>
    terra::crds(df = TRUE) |>
    setDT()
  # (iii) Define containers
  xinit_containers <-
    assemble_xinit_containers(.timeline = timeline,
                              .xinit = list(forward = capture_xy,
                                            backward = recapture_xy),
                              .radius = 750,
                              .mobility = 750,
                              .map = map)

  #### Example (6): Assemble acoustic containers
  # Assemble acoustic containers for the `acoustics` dataset above
  acoustics_containers <- assemble_acoustics_containers(.timeline = timeline,
                                              .acoustics = acoustics,
                                              .mobility = 750,
                                              .map = map)
  # As for assemble_xinit_containers(), this function returns a list:
  summary(acoustics_containers)
  # Use the `forward` element for `pf_filter()` with `.direction = "forward"`
  head(acoustics_containers$forward)
  # Use the `backward` for `pf_filter()` with `.direction = "backward"`
  head(acoustics_containers$backward)

  #### Example (7): Collate containers for multiple datasets
  containers <- assemble_containers(xinit_containers, acoustics_containers)

  #### Example (8): Implement particle filter
  # Use `pf_filter()` to implement the particle filter
  # A list of assembled datasets is passed to the `yobs` argument
  # The corresponding `ModelObs` sub-types must also be specified, e.g.:
  # * `ModelObsAcousticLogisTrunc`
  # * `ModelObsDepthUniformSeabed`
  # * `ModelObsContainer`

}