XY Shifts File Format#

Overview#

The shifts_xy.csv file contains pairwise XY shifts between consecutive slices in a serial sectioning dataset. This file is essential for aligning slices during 3D reconstruction.

File Format#

Location#

Generated by preprocessing pipeline at: {output}/shifts_xy.csv

Structure#

CSV format with header row:

fixed_id,moving_id,x_shift,y_shift,x_shift_mm,y_shift_mm
0,1,156.234,-23.456,0.0234,-0.0035
1,2,142.567,-18.234,0.0214,-0.0027
2,3,161.890,-25.678,0.0243,-0.0039

Columns#

Column	Type	Description
`fixed_id`	int	Slice ID of the fixed (reference) slice
`moving_id`	int	Slice ID of the moving slice
`x_shift`	float	X shift in pixels
`y_shift`	float	Y shift in pixels
`x_shift_mm`	float	X shift in millimeters
`y_shift_mm`	float	Y shift in millimeters

Shift Direction#

The shifts represent the transformation needed to align the moving slice to the fixed slice:

moving_slice_aligned = moving_slice + (x_shift, y_shift)

Or equivalently, the shifts represent the position difference between consecutive slices:

x_shift = mosaic_xmin[fixed] - mosaic_xmin[moving]
y_shift = mosaic_ymin[fixed] - mosaic_ymin[moving]

Generation#

Script#

linum_estimate_xy_shift_from_metadata.py <tiles_directory> <output_file> [--n_processes N]

Process#

Scans tile directory for all slices
Extracts stage positions from tile metadata
Computes mosaic origin (xmin, ymin) for each slice
Calculates pairwise shifts between consecutive slices
Converts to pixels using tile resolution
Writes CSV file

Source Code Reference## From linum_estimate_xy_shift_from_metadata.py

# Compute the shift between slices in mm
x_shifts_mm = []
y_shifts_mm = []
for i in range(n_slices - 1):
    dx = xmin_mm[i] - xmin_mm[i + 1]
    dy = ymin_mm[i] - ymin_mm[i + 1]
    x_shifts_mm.append(dx)
    y_shifts_mm.append(dy)

# Convert the shifts in pixel
x_shift_px = np.array(x_shifts_mm) / tile_resolution[0]
y_shift_px = np.array(y_shifts_mm) / tile_resolution[1]

Usage in Reconstruction#

Common Space Alignment#

The linum_align_mosaics_3d_from_shifts.py script uses shifts to bring all slices into a common coordinate space:

Load shifts: Read all pairwise shifts from CSV
Compute cumulative shifts: Sum shifts from first slice to each subsequent slice
Determine common shape: Find bounding box encompassing all aligned slices
Apply shifts: Translate each slice by its cumulative shift

Cumulative Shift Calculation#

# Cumulative shifts (position relative to first slice)
cumsum = [0]  # First slice has zero offset
for i in range(n_slices - 1):
    cumsum.append(cumsum[i] + shifts[i])

# Example:
# Pairwise shifts: [10, 8, 12, 5]
# Cumulative:      [0, 10, 18, 30, 35]

Handling Missing or Skipped Slices#

Problem#

If some slices are excluded from reconstruction, the pairwise shifts must be accumulated correctly.

Solution#

See SLICE_CONFIG_FEATURE.md for the slice configuration system that handles this.

Example#

Original slices: 0, 1, 2, 3
Shifts: 0→1: 10, 1→2: 8, 2→3: 12

If slice 2 is skipped:

Processing slices: 0, 1, 3
Cumulative for slice 3: 10 + 8 + 12 = 30 (NOT just 12!)

Validation#

Check File Contents#

# View shifts file
cat shifts_xy.csv

# Count entries
wc -l shifts_xy.csv

# Check for NaN or invalid values
grep -E "nan|inf|NaN" shifts_xy.csv

Validate Against Slices#

import pandas as pd

# Load shifts
df = pd.read_csv('shifts_xy.csv')

# Get all slice IDs mentioned
slice_ids = set(df['fixed_id'].tolist() + df['moving_id'].tolist())
print(f"Slices in shifts file: {sorted(slice_ids)}")

# Check for consecutive IDs
expected = set(range(min(slice_ids), max(slice_ids) + 1))
missing = expected - slice_ids
if missing:
    print(f"WARNING: Missing slice IDs: {missing}")

Correcting Erroneous Shifts#

Problem#

The shifts file may contain erroneous large values due to:

Encoder glitch spikes (stage reports a large step that the next step reverses)
Mosaic grid expansion between slices (legacy shifts files where xmin_mm jumps by whole tile columns as tissue grows)
Genuine stage re-homing events (preserved — not an artefact)
Metadata recording errors

Uncorrected, these errors cause slices to drift out of the common volume.

Solution: Re-homing Detection (pipeline default)#

Use linum_detect_rehoming.py (run automatically by the 3D reconstruction pipeline when detect_rehoming = true) to emit a corrected shifts CSV:

linum_detect_rehoming.py shifts_xy.csv shifts_xy_clean.csv \
    --return_fraction 0.4 \
    --max_shift_mm 0.5 \
    --tile_fov_mm 0.875   # only for legacy shifts files

Detection criterion (encoder glitch spike):

|step[i] + step[i±1]| < return_fraction × |step[i]|

i.e. the round-trip magnitude is less than return_fraction times the single- step magnitude (default 0.4 → adjacent step reverses more than 60%). Spike steps are zeroed; genuine re-homing events (large step that stays) are preserved.

The output CSV adds a reliable column (0 when the corrected step is still large or uncertain), consumed downstream by linum_align_mosaics_3d_from_shifts.py --refine_unreliable.

Pipeline Configuration#

In nextflow.config:

params {
    // Re-homing detection (upstream of common-space alignment)
    detect_rehoming         = true
    rehoming_return_fraction = 0.4
    rehoming_max_shift_mm    = 0.5
    tile_fov_mm              = null  // 0.875 only for legacy shifts files

    // Image-based refinement of reliable=0 transitions
    common_space_refine_unreliable          = false
    common_space_refine_max_discrepancy_px  = 0
    common_space_refine_min_correlation     = 0.0
}

Analysing shifts independently#

Use linum_analyze_shifts.py to produce a drift report and outlier plot without modifying the shifts file:

linum_analyze_shifts.py shifts_xy.csv output_dir/ --iqr_multiplier 1.5

Troubleshooting#

Shift Values Too Large#

Symptom: Large pixel shifts (>1000 pixels)
Cause: Stage position metadata may be incorrect or in wrong units
Solution: Check tile metadata; verify stage coordinates are in mm

Inconsistent Shift Signs#

Symptom: Shifts alternate between positive and negative
Cause: Stage movement direction may vary between slices
Solution: This is usually normal; verify reconstructed volume looks correct

Missing Slices in File#

Symptom: Some slice IDs missing from shifts file
Cause: Those slices weren’t found during preprocessing
Solution: Check if raw tiles exist for missing slices; re-run preprocessing

Zero Shifts#

Symptom: All shifts are exactly 0
Cause: Stage positions not available in tile metadata
Solution: Check that tiles have position metadata; use use_stage_positions=True

Related Files#

File	Description
`slice_config.csv`	Controls which slices to include
`mosaic_grid_3d_z*.ome.zarr`	Mosaic grids aligned using these shifts
`3d_volume.ome.zarr`	Final reconstructed volume

Example#

Sample shifts_xy.csv#

fixed_id,moving_id,x_shift,y_shift,x_shift_mm,y_shift_mm
0,1,156.234,-23.456,0.0234,-0.0035
1,2,142.567,-18.234,0.0214,-0.0027
2,3,161.890,-25.678,0.0243,-0.0039
3,4,148.123,-20.567,0.0222,-0.0031
4,5,155.456,-22.890,0.0233,-0.0034

Interpretation#

Slices: 0 through 5 (6 total slices)
Average X shift: ~153 pixels (~0.023 mm)
Average Y shift: ~-22 pixels (~-0.003 mm)
Slices are shifting consistently in negative Y direction