Validation & Diagnostics¶
The validation package provides tools for analyzing calibration quality and reconstruction accuracy.
Reprojection Analysis¶
Reprojection error computation for calibration validation.
- class aquacal.validation.reprojection.ReprojectionErrors(rms, per_camera, per_frame, residuals, num_observations, camera_labels=None)[source]¶
Bases:
objectContainer for reprojection error statistics.
- Parameters:
- residuals¶
(N, 2) array of per-corner residuals (detected - projected)
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.float64]]
- aquacal.validation.reprojection.compute_reprojection_errors(calibration, detections, board_poses)[source]¶
Compute reprojection errors for all observations.
For each frame, camera, and detected corner: 1. Transform corner from board frame to world frame using board pose 2. Project through refractive interface using refractive_project() 3. Compute residual: detected_pixel - projected_pixel
- Parameters:
calibration (CalibrationResult) – Complete calibration result containing camera calibrations and interface parameters
detections (DetectionResult) – Detection result with 2D corner observations
board_poses (dict[int, BoardPose]) – Dict mapping frame_idx to BoardPose (optimized poses)
- Returns:
ReprojectionErrors with all statistics computed
- Return type:
Note
For details on refractive projection, see the Refractive Geometry guide.
Notes
Skips observations where refractive_project() returns None
RMS is sqrt(mean(residual_x^2 + residual_y^2))
- aquacal.validation.reprojection.compute_reprojection_error_single(camera, interface, board, board_pose, detection)[source]¶
Compute reprojection errors for a single camera/frame pair.
- Parameters:
camera (Camera) – Camera object with intrinsics and extrinsics
interface (Interface) – Interface object configured for this camera
board (BoardGeometry) – Board geometry for corner positions
board_pose (BoardPose) – Pose of board for this frame
detection (Detection) – Detected corners in this camera/frame
- Returns:
residuals: (M, 2) array of pixel residuals for valid corners
valid_ids: (M,) array of corner IDs that were successfully projected
Returns (None, None) if no corners could be projected.
- Return type:
Tuple of (residuals, valid_ids)
Reconstruction Validation¶
3D reconstruction metrics using known ChArUco geometry.
- aquacal.validation.reconstruction.get_adjacent_corner_pairs(board)[source]¶
Get pairs of adjacent corners on the board (separated by one square_size).
Adjacent means horizontally or vertically neighboring on the checker grid (not diagonal). Each pair appears once with lower ID first.
- class aquacal.validation.reconstruction.SpatialMeasurements(positions, signed_errors, frame_indices)[source]¶
Bases:
objectPer-measurement spatial data from 3D reconstruction validation.
Each measurement is a distance comparison between two adjacent triangulated board corners. The position is the midpoint of the two corners in world frame.
- Parameters:
- positions¶
(N, 3) array of midpoint positions in world frame (meters)
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.float64]]
- signed_errors¶
(N,) array of signed distance errors (meters). Positive = overestimate, negative = underestimate.
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.float64]]
- frame_indices¶
(N,) array of frame index for each measurement
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.int32]]
- class aquacal.validation.reconstruction.DistanceErrors(mean, std, max_error, num_comparisons, per_corner_pair=None, signed_mean=0.0, rmse=0.0, percent_error=0.0, num_frames=0, spatial=None)[source]¶
Bases:
objectContainer for 3D distance error statistics.
- Parameters:
- per_corner_pair¶
Optional dict mapping (id1, id2) to signed error in meters
- spatial¶
Per-measurement spatial data (optional)
- class aquacal.validation.reconstruction.DepthBinnedErrors(bin_edges, bin_centers, signed_means, signed_stds, counts)[source]¶
Bases:
objectDepth-stratified signed error statistics.
- Parameters:
- bin_edges¶
(B+1,) array of bin edges in meters (Z coordinates)
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.float64]]
- bin_centers¶
(B,) array of bin center Z values in meters
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.float64]]
- signed_means¶
(B,) array of mean signed error per bin (meters)
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.float64]]
- signed_stds¶
(B,) array of std of signed error per bin (meters)
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.float64]]
- counts¶
(B,) array of number of measurements per bin
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.int32]]
- class aquacal.validation.reconstruction.SpatialErrorGrid(depth_bin_edges, x_edges, y_edges, grids, counts)[source]¶
Bases:
objectXY error grids within depth slices.
Each depth slice has a 2D grid of mean signed errors and counts.
- Parameters:
- depth_bin_edges¶
(B+1,) array of depth bin edges in meters
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.float64]]
- x_edges¶
(Gx+1,) array of X grid edges in meters
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.float64]]
- y_edges¶
(Gy+1,) array of Y grid edges in meters
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.float64]]
- grids¶
(B, Gy, Gx) array of mean signed error per cell (meters). NaN where no measurements fall.
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.float64]]
- counts¶
(B, Gy, Gx) array of measurement counts per cell
- Type:
numpy.ndarray[tuple[int, …], numpy.dtype[numpy.int32]]
- aquacal.validation.reconstruction.triangulate_charuco_corners(calibration, detections, frame_idx)[source]¶
Triangulate all ChArUco corners visible in 2+ cameras for a single frame.
- Parameters:
calibration (CalibrationResult) – Complete calibration result
detections (DetectionResult) – Detection result with pixel observations
frame_idx (int) – Frame index to process
- Returns:
Dict mapping corner_id to triangulated 3D position in world frame. Only includes corners visible in at least 2 cameras. Empty dict if frame not in detections or insufficient observations.
- Return type:
- aquacal.validation.reconstruction.compute_3d_distance_errors(calibration, detections, board, include_per_pair=False, include_spatial=False)[source]¶
Compute 3D distance errors by comparing triangulated corner distances to known geometry.
For each frame, triangulates corners visible in 2+ cameras, then compares distances between adjacent corners to the expected square_size. Aggregates statistics across all frames.
- Parameters:
calibration (CalibrationResult) – Complete calibration result
detections (DetectionResult) – Detection result with pixel observations
board (BoardGeometry) – Board geometry with known corner positions
include_per_pair (bool) – If True, populate per_corner_pair in result
include_spatial (bool) – If True, populate spatial field with per-measurement data
- Returns:
DistanceErrors with aggregated statistics across all frames.
- Return type:
Notes
Only compares adjacent corners (horizontally or vertically neighboring)
Only compares corners that were both successfully triangulated
Expected distance is always board.config.square_size for adjacent pairs
Returns DistanceErrors with mean=0, std=0, num_comparisons=0 if no valid pairs
- aquacal.validation.reconstruction.compute_board_planarity_error(triangulated_corners)[source]¶
Compute RMS distance of triangulated corners from best-fit plane.
Fits a plane to the triangulated corners using SVD, then computes the RMS of perpendicular distances from each corner to the plane.
- Parameters:
triangulated_corners (dict[int, ndarray[tuple[int, ...], dtype[float64]]]) – Dict mapping corner_id to 3D position
- Returns:
RMS planarity error in meters, or None if fewer than 3 corners.
- Return type:
float | None
Notes
Requires at least 3 corners to fit a plane
Lower values indicate better triangulation consistency
- aquacal.validation.reconstruction.bin_by_depth(spatial, n_bins=10)[source]¶
Bin spatial measurements by Z coordinate and compute statistics per bin.
- Parameters:
spatial (SpatialMeasurements) – SpatialMeasurements containing positions and signed errors
n_bins (int) – Number of depth bins (default 10)
- Returns:
DepthBinnedErrors with statistics per bin
- Raises:
ValueError – If spatial has zero measurements
- Return type:
Notes
Bins with zero measurements have NaN for signed_means and signed_stds
Bin edges are computed using np.linspace from min to max Z
- aquacal.validation.reconstruction.compute_xy_error_grids(spatial, depth_bin_edges, xy_grid_size=(8, 8), xy_range=None)[source]¶
Bin spatial measurements into XY grids within depth slices.
- Parameters:
spatial (SpatialMeasurements) – SpatialMeasurements containing positions and signed errors
depth_bin_edges (ndarray[tuple[int, ...], dtype[float64]]) – (B+1,) array of depth bin edges in meters (Z coordinates). Use the bin_edges from DepthBinnedErrors to ensure consistency.
xy_grid_size (tuple[int, int]) – Tuple (n_x_bins, n_y_bins) for the XY grid within each depth slice
xy_range (tuple[tuple[float, float], tuple[float, float]] | None) – Optional ((x_min, x_max), (y_min, y_max)). If None, derived from data extent.
- Returns:
SpatialErrorGrid with XY grids for each depth bin
- Raises:
ValueError – If spatial has zero measurements
- Return type:
Notes
For each depth bin, selects measurements whose Z falls within [edge_i, edge_{i+1})
Last bin uses <= for right edge to include boundary values
Within each depth slice, bins by X and Y using np.linspace for edges
Cells with zero measurements get NaN in grids array
- aquacal.validation.reconstruction.save_spatial_measurements(spatial, path)[source]¶
Save spatial measurements to CSV file.
- Parameters:
spatial (SpatialMeasurements) – SpatialMeasurements to save
path (Path) – Output file path
- Return type:
None
Notes
CSV columns: x, y, z, signed_error, frame_idx
One row per measurement
- aquacal.validation.reconstruction.load_spatial_measurements(path)[source]¶
Load spatial measurements from CSV file.
- Parameters:
path (Path) – Path to CSV file created by save_spatial_measurements()
- Returns:
SpatialMeasurements with data from CSV
- Raises:
FileNotFoundError – If path doesn’t exist
- Return type:
Diagnostics¶
Detailed error analysis and diagnostic reporting.
- class aquacal.validation.diagnostics.DiagnosticReport(reprojection, reconstruction, spatial_error_maps, depth_errors, recommendations, summary)[source]¶
Bases:
objectComplete diagnostic report for calibration quality.
- Parameters:
- reprojection¶
Reprojection error statistics
- reconstruction¶
3D reconstruction error statistics
- spatial_error_maps¶
Dict mapping camera name to error heatmap array
- Type:
dict[str, numpy.ndarray[tuple[int, …], numpy.dtype[numpy.float64]]]
- depth_errors¶
DataFrame with depth-stratified error analysis
- Type:
pandas.core.frame.DataFrame
- aquacal.validation.diagnostics.compute_spatial_error_map(reprojection_errors, detections, camera_name, image_size, grid_size=(10, 10))[source]¶
Compute spatial distribution of reprojection errors for one camera.
Bins the image into a grid and computes mean error magnitude in each cell.
- Parameters:
reprojection_errors (ReprojectionErrors) – Pre-computed reprojection errors (needs residuals)
detections (DetectionResult) – Detection result with pixel positions
camera_name (str) – Camera to analyze
- Returns:
2D array of shape (rows, cols) with mean error magnitude per cell. Cells with no observations contain NaN.
- Return type:
Notes
Error magnitude is sqrt(residual_x^2 + residual_y^2)
Useful for identifying spatially-varying calibration issues
- aquacal.validation.diagnostics.compute_depth_stratified_errors(calibration, detections, board_poses, reprojection_errors, board, num_bins=5)[source]¶
Analyze reprojection error as a function of depth below water surface.
- Parameters:
calibration (CalibrationResult) – Complete calibration result
detections (DetectionResult) – Detection result with pixel observations
board_poses (dict[int, BoardPose]) – Dict mapping frame_idx to BoardPose
reprojection_errors (ReprojectionErrors) – Pre-computed reprojection errors
board (BoardGeometry) – Board geometry
num_bins (int) – Number of depth bins
- Returns:
depth_min: Lower bound of depth bin (meters)
depth_max: Upper bound of depth bin (meters)
mean_error: Mean reprojection error in bin (pixels)
std_error: Std deviation of error in bin (pixels)
num_observations: Count of observations in bin
- Return type:
DataFrame with columns
Notes
Depth is measured from water surface (Z - interface_z)
Helps identify depth-dependent calibration issues
- aquacal.validation.diagnostics.compute_camera_heights(calibration)[source]¶
Compute per-camera height above the water surface.
For each camera, computes h_c = water_z - camera_z (positive means camera is above water in Z-down frame).
- Parameters:
calibration (CalibrationResult) – Complete calibration result
- Returns:
“water_z”: The water surface Z-coordinate (from water_z)
”per_camera_height”: Dict mapping camera name to height above water
”mean_height”: Mean camera height above water
”height_spread”: Max - min camera height
- Return type:
Dict with keys
- aquacal.validation.diagnostics.generate_recommendations(reprojection, reconstruction, depth_errors, camera_heights=None, auxiliary_reprojection=None)[source]¶
Generate human-readable recommendations based on diagnostic results.
- Parameters:
reprojection (ReprojectionErrors) – Reprojection error statistics (primary cameras only)
reconstruction (DistanceErrors) – 3D reconstruction error statistics (primary cameras only)
depth_errors (DataFrame) – Depth-stratified error table
camera_heights (dict | None) – Optional camera height data from compute_camera_heights()
auxiliary_reprojection (ReprojectionErrors | None) – Optional reprojection errors for auxiliary cameras
- Returns:
List of recommendation strings.
- Return type:
- Example recommendations:
“Reprojection RMS (0.8 px) is within acceptable range (<1.0 px)”
“Camera ‘cam2’ has elevated error (1.5 px) - check lens/mounting”
“Error increases with depth - consider re-estimating interface”
- aquacal.validation.diagnostics.generate_diagnostic_report(calibration, detections, board_poses, reprojection_errors, reconstruction_errors, board, auxiliary_reprojection=None)[source]¶
Generate complete diagnostic report.
- Parameters:
calibration (CalibrationResult) – Complete calibration result (primary cameras only for summary stats)
detections (DetectionResult) – Detection result
board_poses (dict[int, BoardPose]) – Dict mapping frame_idx to BoardPose
reprojection_errors (ReprojectionErrors) – Pre-computed from reprojection.py (primary cameras only)
reconstruction_errors (DistanceErrors) – Pre-computed from reconstruction.py (primary cameras only)
board (BoardGeometry) – Board geometry
auxiliary_reprojection (ReprojectionErrors | None) – Optional reprojection errors for auxiliary cameras
- Returns:
DiagnosticReport with all analysis results.
- Return type:
- aquacal.validation.diagnostics.plot_camera_rig(calibration, arrow_length=0.05, title='Camera Rig Layout')[source]¶
Plot 3D camera rig with positions, orientations, and water surface in three viewing angles.
- Parameters:
calibration (CalibrationResult) – CalibrationResult with camera extrinsics and interface distances
arrow_length (float) – Length of optical axis arrows in world units (meters)
title (str) – Overall figure title
- Returns:
matplotlib Figure with three subplots (perspective, top-down, side view)
- Return type:
plt.Figure
- aquacal.validation.diagnostics.plot_reprojection_quiver(calibration, detections, reprojection_errors, camera_name, ax=None, scale=1.0)[source]¶
Plot reprojection error quiver for one camera.
- Parameters:
calibration (CalibrationResult) – CalibrationResult (for image_size)
detections (DetectionResult) – DetectionResult with pixel positions
reprojection_errors (ReprojectionErrors) – Pre-computed errors with residuals array
camera_name (str) – Which camera to plot
ax (plt.Axes | None) – Optional existing axes. If None, creates new figure.
scale (float) – Arrow scale factor (1.0 = true pixel scale, >1 exaggerates for visibility)
- Returns:
matplotlib Figure
- Return type:
plt.Figure
- aquacal.validation.diagnostics.plot_per_camera_error(result)[source]¶
Bar chart of per-camera RMS reprojection error with overall RMS line.
- Parameters:
result (CalibrationResult) – CalibrationResult with populated diagnostics.
- Returns:
matplotlib Figure.
- Return type:
plt.Figure
- aquacal.validation.diagnostics.plot_error_distribution(reprojection_errors, auxiliary_cameras=None)[source]¶
Histogram of per-corner reprojection error magnitudes.
When auxiliary_cameras is provided and camera labels are available, draws separate normalised distributions for primary and auxiliary cameras.
- Parameters:
reprojection_errors (ReprojectionErrors) – Pre-computed reprojection errors with residuals.
auxiliary_cameras (set[str] | None) – Set of auxiliary camera names. When provided with camera labels, the plot shows separate distributions.
- Returns:
matplotlib Figure.
- Return type:
plt.Figure
- aquacal.validation.diagnostics.save_diagnostic_report(report, calibration, detections, output_dir, save_images=True, auxiliary_reprojection=None, timings=None)[source]¶
Save diagnostic report to disk.
Creates: - diagnostics.json: Summary statistics and recommendations - spatial_error_{cam}.png: Per-camera error heatmaps (if save_images=True) - camera_rig.png: 3D camera rig visualization (if save_images=True) - quiver_{cam}.png: Per-camera reprojection error quiver plots (if save_images=True) - depth_errors.csv: Depth-stratified error table
- Parameters:
report (DiagnosticReport) – DiagnosticReport to save (contains primary-only stats)
calibration (CalibrationResult) – CalibrationResult for camera rig and quiver plots (full result with all cameras)
detections (DetectionResult) – DetectionResult for quiver plots
output_dir (Path) – Directory for output files (created if doesn’t exist)
save_images (bool) – Whether to render and save images
auxiliary_reprojection (ReprojectionErrors | None) – Optional reprojection errors for auxiliary cameras
timings (dict[str, object] | None) – Optional pre-shaped timings payload (e.g.,
{"seconds_per_stage": {...}, "total_seconds": 123.4}) to embed under the top-level"timings"key indiagnostics.json. WhenNone, the key is omitted (backward compatible).
- Returns:
“json”: Path to diagnostics.json
”csv”: Path to depth_errors.csv
”images”: Dict of camera_name -> spatial error image path (if save_images=True)
”rig”: Path to camera_rig.png (if save_images=True)
”quiver”: Dict of camera_name -> quiver image path (if save_images=True)
- Return type:
Dict mapping output type to file path
Comparison Tools¶
Cross-run calibration comparison.
- class aquacal.validation.comparison.ComparisonResult(labels, metric_table, per_camera_metrics, parameter_diffs)[source]¶
Bases:
objectStructured comparison of N calibration runs.
- Parameters:
- metric_table¶
DataFrame with rows = runs (indexed by label), columns = quality metrics
- Type:
pandas.core.frame.DataFrame
- per_camera_metrics¶
Dict mapping camera_name -> DataFrame with rows = runs, columns = per-camera metrics
- parameter_diffs¶
DataFrame with pairwise parameter differences between runs
- Type:
pandas.core.frame.DataFrame
- aquacal.validation.comparison.compare_calibrations(results, labels)[source]¶
Compare N calibration runs and return structured differences.
- Parameters:
results (list[CalibrationResult]) – List of calibration results to compare (must have len >= 2)
labels (list[str]) – User-assigned labels for each run (must be unique)
- Returns:
ComparisonResult containing metric tables and parameter differences
- Raises:
ValueError – If len(results) < 2, len(results) != len(labels), or labels not unique
- Return type:
- aquacal.validation.comparison.write_comparison_report(comparison, results, output_dir, save_plots=True, depth_data=None, spatial_data=None)[source]¶
Write comparison report to disk as CSV tables and PNG plots.
- Parameters:
comparison (ComparisonResult) – ComparisonResult from compare_calibrations()
results (list[CalibrationResult]) – List of CalibrationResult objects used to generate comparison
output_dir (str | Path) – Directory for output files (created if doesn’t exist)
save_plots (bool) – Whether to generate and save PNG plots
depth_data (dict[str, DepthBinnedErrors] | None) – Optional dict mapping run label -> DepthBinnedErrors for depth plot
spatial_data (dict[str, SpatialMeasurements] | None) – Optional dict mapping run label -> SpatialMeasurements for XY heatmaps
- Returns:
“metrics_csv”: metrics_summary.csv
”per_camera_csv”: per_camera_metrics.csv
”parameter_diffs_csv”: parameter_diffs.csv
”rms_bar_chart”: rms_bar_chart.png (if save_plots=True)
”position_overlay”: position_overlay.png (if save_plots=True)
”z_position_dumbbell”: z_position_dumbbell.png (if save_plots=True)
”depth_error_plot”: depth_error_comparison.png (if depth_data and save_plots=True)
”depth_binned_csv”: depth_binned_errors.csv (if depth_data)
”xy_error_heatmaps”: xy_error_heatmaps.png (if spatial_data and save_plots=True)
- Return type:
Dict mapping output type to file path
- aquacal.validation.comparison.plot_rms_bar_chart(comparison)[source]¶
Create grouped bar chart of per-camera reprojection RMS.
- Parameters:
comparison (ComparisonResult) – ComparisonResult with per_camera_metrics
- Returns:
matplotlib Figure object
- aquacal.validation.comparison.plot_position_overlay(comparison, results)[source]¶
Create 2D top-down scatter plot of camera positions.
- Parameters:
comparison (ComparisonResult) – ComparisonResult with labels
results (list[CalibrationResult]) – List of CalibrationResult objects
- Returns:
matplotlib Figure object
- aquacal.validation.comparison.plot_z_position_dumbbell(comparison)[source]¶
Create dumbbell chart of camera Z positions across runs.
For each camera (sorted alphabetically), plots one dot per run at its Z position, connected by a horizontal line. Cameras missing from a run are skipped.
- Parameters:
comparison (ComparisonResult) – ComparisonResult with per_camera_metrics
- Returns:
matplotlib Figure object
- aquacal.validation.comparison.plot_xy_error_heatmaps(grids)[source]¶
Create heatmap grid of XY error distributions across depth slices.
- Parameters:
grids (dict[str, SpatialErrorGrid]) – Dict mapping run label -> SpatialErrorGrid
- Returns:
matplotlib Figure object
Notes
Subplot grid: rows = runs (sorted by label), columns = depth bins
Each subplot shows mean signed error in mm using diverging colormap
Symmetric color scale centered at zero across all runs
Depth bins with zero measurements across ALL runs are skipped
- aquacal.validation.comparison.plot_depth_error_comparison(depth_data)[source]¶
Create line plot of depth-stratified signed error across runs.
- Parameters:
depth_data (dict[str, DepthBinnedErrors]) – Dict mapping run label -> DepthBinnedErrors
- Returns:
matplotlib Figure object
Notes
X-axis: depth (Z position in meters)
Y-axis: signed mean error in millimeters
One line per run with legend
Bins with zero counts are skipped (NaN values create gaps)
Optional shaded region showing +/- 1 std per run