Introduction

Rustpix is a high-performance pixel detector data processing library for neutron imaging. It processes Timepix3 (TPX3) data with throughput exceeding 96 million hits per second, featuring multiple clustering algorithms, centroid extraction, and Python bindings.

Features

• Fast TPX3 Processing: Parallel packet parsing with memory-mapped I/O
• Multiple Clustering Algorithms:
  • ABS (Adjacency-Based Search) - 8-connectivity clustering
  • DBSCAN - Density-based with spatial indexing
  • Grid - Parallel grid-based clustering
• Streaming Architecture: Process files larger than RAM
• Python Bindings: Thin wrappers with NumPy integration
• CLI Tool: Command-line interface for batch processing
• GUI Application: Interactive analysis with real-time visualization
• Multiple Output Formats: HDF5, Arrow, CSV

Performance

• Throughput: 96M+ hits/sec on modern hardware
• Memory: Streaming architecture processes files larger than RAM
• Parallel: Multi-threaded clustering with rayon
• Optimized: SIMD-friendly data layouts (SoA)

Getting Started

Choose the interface that best fits your workflow:

• Python API - For scripting and integration with scientific Python stack
• CLI Tool - For batch processing and shell scripts
• GUI Application - For interactive exploration and analysis

Workspace Structure

Rustpix is organized as a Rust workspace with multiple crates:

Links

• GitHub Repository
• PyPI Package
• crates.io (rustpix-core)
• Rust API Documentation

Installation

Rustpix can be installed in several ways depending on your needs:

Quick Install

Python Users

pip install rustpix

macOS Users (GUI)

brew tap ornlneutronimaging/rustpix
brew install --cask rustpix

Rust Users (CLI)

cargo install rustpix-cli

System Requirements

• Python: 3.11 or later
• macOS: Big Sur (11.0) or later, Apple Silicon (ARM64)
• Rust: 1.70 or later (for building from source)

Python Installation

pip

The recommended way to install rustpix for Python is via pip:

pip install rustpix

This installs pre-built wheels for:

• Linux (x86_64, glibc 2.28+)
• macOS (ARM64 and x86_64)
• Windows (x86_64)

Verify Installation

import rustpix
print(rustpix.__version__)

Virtual Environment (Recommended)

We recommend using a virtual environment:

# Create environment
python -m venv .venv
source .venv/bin/activate  # Linux/macOS
# or: .venv\Scripts\activate  # Windows

# Install
pip install rustpix

With Scientific Stack

For data analysis workflows, install alongside NumPy and other tools:

pip install rustpix numpy matplotlib h5py

Jupyter Notebooks

Rustpix works well in Jupyter notebooks:

pip install rustpix jupyterlab
jupyter lab

Troubleshooting

Python Version

Rustpix requires Python 3.11 or later. Check your version:

python --version

Wheels Not Available

If no wheel is available for your platform, pip will attempt to build from source, which requires Rust. See From Source for build instructions.

macOS Installation (Homebrew)

GUI Application

Install the rustpix GUI application via Homebrew:

# Add the tap
brew tap ornlneutronimaging/rustpix

# Install the GUI app
brew install --cask rustpix

Launch

After installation, launch from:

• Spotlight: Search for "Rustpix"
• Applications: Find Rustpix in /Applications
• Terminal: open -a Rustpix

Requirements

• macOS Big Sur (11.0) or later
• Apple Silicon (ARM64) architecture

Note: Intel Mac support is available via the CLI tool or Python package.

Updating

brew upgrade --cask rustpix

Uninstalling

brew uninstall --cask rustpix

Gatekeeper Notice

On first launch, macOS may show a security warning. The Homebrew installation automatically handles the quarantine attribute, but if you see a warning:

1. Go to System Preferences > Security & Privacy
2. Click Open Anyway for Rustpix

Rust Installation (cargo)

CLI Tool

Install the command-line interface via cargo:

cargo install rustpix-cli

This installs the rustpix binary to ~/.cargo/bin/.

Verify Installation

rustpix --version
rustpix --help

Library Usage

Add rustpix crates to your Rust project:

# Core types and traits
cargo add rustpix-core

# Clustering algorithms
cargo add rustpix-algorithms

# TPX3 parsing
cargo add rustpix-tpx

# File I/O
cargo add rustpix-io

Example Cargo.toml

[dependencies]
rustpix-core = "1.0"
rustpix-algorithms = "1.0"
rustpix-tpx = "1.0"
rustpix-io = "1.0"

API Documentation

Rust API documentation is available on docs.rs:

• rustpix-core
• rustpix-algorithms
• rustpix-tpx
• rustpix-io

Requirements

• Rust 1.70 or later
• For HDF5 support: HDF5 libraries (automatically handled via static linking)

Building From Source

Prerequisites

• Git
• Rust 1.70+ (install via rustup)
• Python 3.11+ (for Python bindings)
• Pixi (recommended) or Cargo

Clone Repository

git clone https://github.com/ornlneutronimaging/rustpix
cd rustpix

Using Pixi (Recommended)

Pixi manages all dependencies automatically:

# Install pixi
curl -fsSL https://pixi.sh/install.sh | bash

# Install dependencies and build
pixi install
pixi run build

Available Tasks

pixi run test        # Run all tests
pixi run clippy      # Run linter
pixi run gui         # Launch GUI (release mode)
pixi run gui-debug   # Launch GUI (debug mode)
pixi run docs        # Build Rust documentation

Using Cargo

Build without pixi:

# Build all crates
cargo build --release --workspace

# Run tests
cargo test --workspace

# Build CLI
cargo build --release -p rustpix-cli

# Build GUI
cargo build --release -p rustpix-gui

Python Bindings

Build Python package with maturin:

# Install maturin
pip install maturin

# Build and install in development mode
cd rustpix-python
maturin develop --release

Or build a wheel:

maturin build --release -m rustpix-python/Cargo.toml
pip install target/wheels/rustpix-*.whl

Development Setup

For development with hot reloading:

pixi install
pixi run gui-debug  # Faster builds, debug symbols

Troubleshooting

HDF5 Linking Errors

HDF5 is statically linked by default. If you encounter issues:

# Ensure HDF5 dev packages are installed
# Ubuntu/Debian:
sudo apt install libhdf5-dev

# macOS:
brew install hdf5

Python Not Found

Ensure Python 3.11+ is available:

python3 --version
# or with pixi:
pixi run python --version

Python API

The rustpix Python package provides thin wrappers around the high-performance Rust core. Data is returned as NumPy arrays or PyArrow Tables for seamless integration with the scientific Python ecosystem.

Overview

Data Types

HitBatch

Contains raw detector hits with the following fields:

Note: The tof field is stored in 25ns tick units for efficiency. To convert to nanoseconds: tof_ns = tof * 25

NeutronBatch

Contains processed neutron events:

Note: The tof field is stored in 25ns tick units. To convert to nanoseconds: tof_ns = tof * 25

Output Formats

Both HitBatch and NeutronBatch support:

# Convert to NumPy dict of arrays
data = batch.to_numpy()

# Convert to PyArrow Table (requires pyarrow)
table = batch.to_arrow()

Algorithms

Three clustering algorithms are available:

Specify with the algorithm keyword argument:

neutrons = rustpix.process_tpx3_neutrons(
    "data.tpx3",
    algorithm="abs"  # or "dbscan", "grid"
)

Next Steps

• Quick Start - Basic usage examples
• Configuration - Detailed configuration options

Quick Start

Reading Hits

Load all hits from a TPX3 file into memory:

import rustpix

# Read all hits
hits = rustpix.read_tpx3_hits("data.tpx3")

# Convert to NumPy arrays
data = hits.to_numpy()
print(f"Loaded {len(data['x'])} hits")

# Access individual arrays
x = data['x']      # uint16
y = data['y']      # uint16
tof = data['tof']  # uint32, 25ns ticks (multiply by 25 for nanoseconds)
tot = data['tot']  # uint16

Streaming Hits

For large files, stream hits in batches:

import rustpix

for batch in rustpix.stream_tpx3_hits("large_data.tpx3"):
    data = batch.to_numpy()
    process_batch(data)

Processing Neutrons

Convert hits to neutron events using clustering:

import rustpix

# Configure clustering
clustering = rustpix.ClusteringConfig(
    radius=5.0,              # spatial epsilon (pixels)
    temporal_window_ns=75.0, # temporal epsilon (nanoseconds)
    min_cluster_size=1
)

# Configure centroid extraction
extraction = rustpix.ExtractionConfig(
    super_resolution_factor=8.0,
    weighted_by_tot=True,
    min_tot_threshold=10
)

# Process file (returns single batch)
neutrons = rustpix.process_tpx3_neutrons(
    "data.tpx3",
    clustering_config=clustering,
    extraction_config=extraction,
    algorithm="abs",
    collect=True
)

# Convert to NumPy
data = neutrons.to_numpy()
print(f"Found {len(data['x'])} neutron events")

Streaming Neutrons

Stream neutron events for large files:

import rustpix

clustering = rustpix.ClusteringConfig(radius=5.0, temporal_window_ns=75.0)

# Stream neutrons (default mode)
for batch in rustpix.stream_tpx3_neutrons(
    "large_data.tpx3",
    clustering_config=clustering
):
    data = batch.to_numpy()
    save_batch(data)

Or use process_tpx3_neutrons without collect=True:

# Streaming is the default
for batch in rustpix.process_tpx3_neutrons(
    "large_data.tpx3",
    clustering_config=clustering
):
    process_batch(batch.to_numpy())

Clustering Hits

Cluster an existing HitBatch:

import rustpix

# Read hits
hits = rustpix.read_tpx3_hits("data.tpx3")

# Cluster
clustering = rustpix.ClusteringConfig(radius=5.0, temporal_window_ns=75.0)
neutrons = rustpix.cluster_hits(
    hits,
    clustering_config=clustering,
    algorithm="dbscan"
)

data = neutrons.to_numpy()

PyArrow Integration

Export to PyArrow for Parquet, Arrow IPC, or DataFrame conversion:

import rustpix

neutrons = rustpix.process_tpx3_neutrons("data.tpx3", collect=True)

# Convert to PyArrow Table
table = neutrons.to_arrow()

# Save as Parquet
import pyarrow.parquet as pq
pq.write_table(table, "neutrons.parquet")

# Convert to Pandas
df = table.to_pandas()

VENUS Detector Defaults

For VENUS detector at SNS:

import rustpix

# Use VENUS-specific defaults
detector = rustpix.DetectorConfig.venus_defaults()
clustering = rustpix.ClusteringConfig.venus_defaults()
extraction = rustpix.ExtractionConfig.venus_defaults()

neutrons = rustpix.process_tpx3_neutrons(
    "venus_data.tpx3",
    detector_config=detector,
    clustering_config=clustering,
    extraction_config=extraction,
    collect=True
)

Out-of-Core Processing

For files larger than RAM:

import rustpix

# Configure memory-bounded processing
for batch in rustpix.stream_tpx3_neutrons(
    "huge_file.tpx3",
    clustering_config=rustpix.ClusteringConfig(),
    memory_fraction=0.5,    # Use up to 50% of RAM
    parallelism=4,          # Worker threads
    async_io=True           # Async reader pipeline
):
    save_batch(batch.to_numpy())

Configuration

DetectorConfig

Configure detector-specific parameters:

import rustpix

config = rustpix.DetectorConfig(
    tdc_frequency_hz=60.0,            # TDC frequency (Hz)
    enable_missing_tdc_correction=True,
    chip_size_x=256,                  # Chip width in pixels
    chip_size_y=256,                  # Chip height in pixels
    chip_transforms=None              # Custom chip transformations
)

Parameters

Chip Transforms

Chip transforms are 2x2 affine matrices plus translation:

# Transform tuple: (a, b, c, d, tx, ty)
# x' = a*x + b*y + tx
# y' = c*x + d*y + ty

config = rustpix.DetectorConfig(
    chip_transforms=[
        (1, 0, 0, 1, 0, 0),      # Identity for chip 0
        (1, 0, 0, 1, 256, 0),    # Chip 1 offset by 256 in X
    ]
)

Presets

# VENUS detector defaults
config = rustpix.DetectorConfig.venus_defaults()

# Load from JSON
config = rustpix.DetectorConfig.from_json('{"tdc_frequency_hz": 60.0}')

ClusteringConfig

Configure the clustering algorithm:

config = rustpix.ClusteringConfig(
    radius=5.0,
    temporal_window_ns=75.0,
    min_cluster_size=1,
    max_cluster_size=None
)

Parameters

Tuning Tips

• radius: Larger values merge more hits. Start with 5.0 for typical neutron events.
• temporal_window_ns: Should match detector timing characteristics. 75ns works for most TPX3 setups.
• min_cluster_size: Set to 2+ to filter noise (single-hit events).
• max_cluster_size: Set to filter large background events (e.g., gamma showers).

ExtractionConfig

Configure centroid extraction:

config = rustpix.ExtractionConfig(
    super_resolution_factor=8.0,
    weighted_by_tot=True,
    min_tot_threshold=10
)

Parameters

Super Resolution

The super_resolution_factor controls sub-pixel precision:

• 1.0: Integer pixel coordinates
• 8.0: 1/8 pixel precision (default)
• 16.0: 1/16 pixel precision

ToT Weighting

When weighted_by_tot=True, the centroid is computed as:

x_centroid = Σ(x_i * tot_i) / Σ(tot_i)
y_centroid = Σ(y_i * tot_i) / Σ(tot_i)

This improves resolution by weighting toward hits with higher charge deposition.

Algorithm-Specific Parameters

Pass algorithm parameters as keyword arguments:

# ABS algorithm
neutrons = rustpix.process_tpx3_neutrons(
    "data.tpx3",
    algorithm="abs",
    abs_scan_interval=1000,  # Scan interval for ABS
    collect=True
)

# DBSCAN algorithm
neutrons = rustpix.process_tpx3_neutrons(
    "data.tpx3",
    algorithm="dbscan",
    dbscan_min_points=2,  # Min points for core sample
    collect=True
)

# Grid algorithm
neutrons = rustpix.process_tpx3_neutrons(
    "data.tpx3",
    algorithm="grid",
    grid_cell_size=10,  # Grid cell size in pixels
    collect=True
)

Out-of-Core Processing Parameters

For streaming with memory constraints:

for batch in rustpix.stream_tpx3_neutrons(
    "huge_file.tpx3",
    clustering_config=rustpix.ClusteringConfig(),

    # Memory management
    out_of_core=True,           # Enable (default: True for streaming)
    memory_fraction=0.5,        # Target 50% of available RAM
    memory_budget_bytes=4_000_000_000,  # Or explicit 4GB limit

    # Parallelism
    parallelism=4,              # Worker threads
    queue_depth=8,              # Pipeline queue depth
    async_io=True               # Async I/O pipeline
):
    process_batch(batch)

Command Line Interface

The rustpix CLI provides batch processing capabilities for TPX3 files.

Installation

cargo install rustpix-cli

Or build from source:

cargo build --release -p rustpix-cli

Commands

Quick Examples

# Process a file
rustpix process input.tpx3 -o output.csv

# Show file info
rustpix info input.tpx3

# Benchmark algorithms
rustpix benchmark input.tpx3

# Get help
rustpix --help
rustpix process --help

Output Formats

The output format is determined by file extension:

See Commands Reference for detailed usage.

Commands Reference

rustpix process

Process TPX3 files to extract neutron events.

rustpix process [OPTIONS] -o <OUTPUT> <INPUT>...

Arguments

Options

Examples

# Basic processing
rustpix process input.tpx3 -o output.csv

# Process multiple files
rustpix process file1.tpx3 file2.tpx3 -o combined.csv

# Use DBSCAN with custom parameters
rustpix process input.tpx3 -o output.csv \
    --algorithm dbscan \
    --radius 3.0 \
    --temporal-window-ns 50.0

# Verbose output with parallel processing
rustpix process input.tpx3 -o output.bin \
    --verbose \
    --parallelism 8 \
    --async-io true

# Memory-constrained processing
rustpix process huge_file.tpx3 -o output.csv \
    --memory-fraction 0.3 \
    --out-of-core true

rustpix info

Display information about a TPX3 file.

rustpix info <INPUT>

Example

$ rustpix info data.tpx3
File: data.tpx3
Size: 104857600 bytes (104.86 MB)
Packets: 6553600
Hits: 5242880
TOF range: 0 - 16666666
X range: 0 - 511
Y range: 0 - 511

rustpix benchmark

Benchmark clustering algorithms on a TPX3 file.

rustpix benchmark [OPTIONS] <INPUT>

Options

Example

$ rustpix benchmark data.tpx3 --iterations 5
Benchmarking with 5242880 hits, 5 iterations
Algorithm  | Mean Time (ms)  | Min Time (ms)   | Max Time (ms)
-----------------------------------------------------------------
ABS        | 245.32          | 238.45          | 256.78
DBSCAN     | 1234.56         | 1198.23         | 1287.34
Grid       | 312.45          | 298.12          | 334.56

rustpix out-of-core-benchmark

Benchmark out-of-core processing modes.

rustpix out-of-core-benchmark [OPTIONS] <INPUT>

Options

Example

$ rustpix out-of-core-benchmark data.tpx3 --parallelism 4 --async-io true
Out-of-core benchmark (3 iterations)
Single-thread avg: 12.345s
Multi-thread avg: 4.567s (threads: 4, async: true)
Speedup: 2.70x

Environment Variables

The CLI respects standard environment variables:

GUI Application

The rustpix GUI provides interactive visualization and analysis of TPX3 data.

Installation

macOS (Homebrew)

brew tap ornlneutronimaging/rustpix
brew install --cask rustpix

From Source

# Using pixi
pixi run gui

# Using cargo
cargo run --release -p rustpix-gui

Features

• Interactive file loading: Open TPX3 files via file dialog or drag-and-drop
• Real-time visualization: View hits and neutron events on 2D detector maps
• Algorithm selection: Choose between ABS, DBSCAN, and Grid clustering
• Parameter tuning: Adjust clustering parameters with immediate visual feedback
• ROI selection: Define regions of interest for focused analysis
• Export options: Save processed data to HDF5, CSV, TIFF, and other formats
• Memory monitoring: Track memory usage during processing

Launching

macOS

• Spotlight: Search for "Rustpix"
• Applications: Find in /Applications
• Terminal: open -a Rustpix

From Source

# Release mode (faster)
pixi run gui

# Debug mode (faster compilation)
pixi run gui-debug

Workflow

1. Load Data

1. Click File > Open or drag a .tpx3 file onto the window
2. Wait for the file to load (progress shown in status bar)
3. Raw hits appear in the visualization panel

2. Configure Processing

1. Select clustering algorithm from the dropdown
2. Adjust parameters:
   • Radius: Spatial clustering distance (pixels)
   • Temporal Window: Time clustering window (nanoseconds)
   • Min Cluster Size: Filter small clusters

3. Process

1. Click Process to run clustering
2. Neutron events appear in the visualization
3. Statistics shown in the info panel

4. Analyze

• Pan/Zoom: Mouse wheel and drag to navigate
• ROI: Draw regions of interest for statistics
• Histogram: View ToF and spatial distributions

5. Export

1. Click File > Export
2. Choose format:
   • HDF5: Full data with metadata
   • CSV: Simple tabular export
   • TIFF: Image export
3. Select output location

Keyboard Shortcuts

System Requirements

• macOS Big Sur (11.0) or later
• Apple Silicon (ARM64) recommended
• 8GB RAM minimum (16GB recommended for large files)
• OpenGL 3.3 or later

Troubleshooting

App Won't Open (macOS)

If macOS blocks the app:

1. Go to System Preferences > Security & Privacy
2. Click Open Anyway for Rustpix

Out of Memory

For very large files:

• Enable streaming mode in preferences
• Reduce the loaded time range
• Use the CLI for batch processing instead

Slow Rendering

• Reduce the number of displayed points (use downsampling)
• Close other applications to free GPU memory
• Try the CLI for processing, GUI for visualization only

Clustering Algorithms

Rustpix provides three clustering algorithms for grouping detector hits into neutron events. Each algorithm has different performance characteristics and is suited for different use cases.

Overview

ABS (Adjacency-Based Search)

The default algorithm. Uses 8-connectivity search to find adjacent pixels within temporal and spatial thresholds.

How It Works

1. Hits are processed in time order
2. For each hit, search for neighbors within radius and temporal window
3. Group connected hits into clusters using flood-fill
4. Periodically scan for completed clusters (configurable interval)

Parameters

When to Use

• General-purpose neutron imaging
• Files with moderate noise levels
• When processing speed is important

neutrons = rustpix.process_tpx3_neutrons(
    "data.tpx3",
    algorithm="abs",
    abs_scan_interval=1000,
    collect=True
)

DBSCAN

Density-Based Spatial Clustering of Applications with Noise. Groups points based on density reachability.

How It Works

1. Build spatial index of all hits
2. For each unvisited hit, find neighbors within epsilon
3. If enough neighbors (min_points), start a cluster
4. Recursively expand cluster with density-reachable points
5. Points not in any cluster are marked as noise

Parameters

When to Use

• High noise environments
• When cluster shape is irregular
• When you need to identify noise points

neutrons = rustpix.process_tpx3_neutrons(
    "data.tpx3",
    algorithm="dbscan",
    dbscan_min_points=2,
    collect=True
)

Grid

Parallel grid-based clustering with spatial indexing.

How It Works

1. Divide detector space into cells
2. Assign hits to cells based on position
3. Process cells in parallel using rayon
4. Merge clusters that span cell boundaries
5. Use union-find for efficient cluster merging

Parameters

When to Use

• Very large datasets
• Multi-core systems
• When throughput is critical

neutrons = rustpix.process_tpx3_neutrons(
    "data.tpx3",
    algorithm="grid",
    grid_cell_size=32,
    collect=True
)

Performance Comparison

Benchmark results on a typical neutron imaging dataset (5M hits):

Choosing an Algorithm

Start with ABS (default)
    │
    ├─ Too much noise? → Try DBSCAN
    │
    ├─ Need more speed? → Try Grid
    │   └─ (especially on multi-core systems)
    │
    └─ Results look good? → Stick with ABS

Parameter Tuning

Spatial Radius

• Too small: Clusters split into multiple events
• Too large: Separate events merged together
• Start with: 5.0 pixels, adjust based on results

Temporal Window

• Too small: Events spanning multiple TDC cycles split
• Too large: Unrelated events merged
• Start with: 75.0 ns (matches typical TPX3 timing)

Min Cluster Size

• 1: Accept all clusters (including noise)
• 2+: Filter single-hit noise events
• Typical: 1-3 depending on noise level

Technical Reference

This section provides detailed technical specifications for rustpix.

Contents

• HDF5 Schema - NeXus-compatible HDF5 file format specification

API Documentation

Rust API

Comprehensive Rust API documentation is available on docs.rs:

• rustpix-core - Core traits and types
• rustpix-algorithms - Clustering algorithms
• rustpix-tpx - TPX3 parser
• rustpix-io - File I/O

Python API

See the Python API chapter for comprehensive Python documentation.

Data Formats

Input: TPX3

Rustpix reads Timepix3 (TPX3) binary files. TPX3 files contain:

• Hit packets (pixel coordinates, timestamp, ToT)
• TDC packets (timing reference)
• Metadata headers

Output Formats

Performance Characteristics

Throughput

Memory Usage

• Streaming mode: Bounded memory, configurable via memory_fraction
• Batch mode: ~100 bytes per hit for full processing
• HDF5 export: Chunked writes, minimal peak memory

Version Compatibility

HDF5 Schema

This document defines the on-disk HDF5 layout for rustpix event data and histograms. The schema is designed for scipp compatibility via NeXus, using NXevent_data for events and NXdata for histograms.

Goals

• Bounded-memory processing for large TPX3 datasets
• scipp-compatible layout via NeXus (NXevent_data + NXdata)
• Clear units and metadata to support TOF ↔ eV conversion
• Optional fields (tot, chip_id, cluster_id) are truly optional

File Structure

/
  rustpix_format_version = "0.1"
  entry/                     (NXentry)
    hits/                    (NXevent_data) [optional]
    neutrons/                (NXevent_data) [optional]
    histogram/               (NXdata)       [optional]
    metadata/                (group)        [optional]

File-Level Conventions

• Root group has attribute: rustpix_format_version = "0.1"
• All groups use NX_class attributes where applicable
• Units are stored as dataset attributes: units = "ns", "pixel", "deg", etc.
• Endianness is native (HDF5 handles portability)

Event Data (NXevent_data)

Event groups follow the NeXus NXevent_data base class, used by SNS/ISIS event files and expected by Mantid.

Required Datasets

Pulse Indexing (for pulsed sources)

Optional Datasets

Cluster ID Convention

• cluster_id >= 0: Valid cluster index
• cluster_id = -1: Unclustered / noise

Event ID Mapping

For imaging data, event_id maps to pixel coordinates:

event_id = y * x_size + x

Group attributes x_size and y_size define the detector dimensions.

Histogram Data (NXdata)

Histogram data is stored in a single NXdata group named histogram.

Group Attributes

NX_class = "NXdata"
signal = "counts"
axes = ["rot_angle", "y", "x", "time_of_flight"]
rot_angle_indices = 0
y_indices = 1
x_indices = 2
time_of_flight_indices = 3

Required Datasets

Axis Representation

• Centers: axis length = N, axis_mode = "centers"
• Edges: axis length = N+1, axis_mode = "edges"

Optional Energy Axis

If flight_path_m and tof_offset_ns are provided:

Conversion Metadata

Stored as attributes at /entry:

TOF to Energy Conversion

Using the non-relativistic relation:

E = (m_n / 2) * (L / t)²

where:
  t = (event_time_offset + tof_offset_ns) * 1e-9  [seconds]
  L = flight_path_m  [meters]
  m_n = neutron mass

Metadata Group

/entry/metadata may contain:

• Detector config (chip transforms, pixel size)
• Clustering config
• Extraction config
• Processing provenance (git sha, rustpix version)
• Instrument context (facility, run ID)

Preferred storage: UTF-8 string dataset named metadata_json.

Implementation Notes

Chunking Strategy

• Events: Chunk along event dimension, 50k–200k events per chunk
• Histograms: Chunk along slowest-changing dimensions (e.g., rot_angle)

Compression

• Start with gzip level 1–4 for balanced I/O
• Use shuffle + compression for integer datasets

Data Types

• Use u64 for timestamps in ns to prevent overflow
• Use f64 for coordinates requiring sub-pixel precision