CUFormer: Modality-Aware CT-Ultrasound Point Cloud Registration for Orthopedic Navigation

arXiv preprint

Lixing Tan¹ Hongye Hao¹ Tianrun Gu¹ Qing Xia^1,* Yuting Guo² Shuai Li¹ Aimin Hao¹

¹Beihang University ²Beijing Information Science and Technology University

^*Corresponding author: xiaqing@buaa.edu.cn

Teaser: preoperative CT and the reconstructed ultrasound point cloud from tracked B-mode images enter CUFormer; the registered CT with ultrasound frames and CT contours on three B-mode images; registered results on fibula, foot, femur and spine.

CUFormer registers intraoperative US bone surfaces to preoperative CT for radiation-free orthopedic navigation. Tracked B-mode images are reconstructed into a partial point cloud and aligned with the CT model. CT-US results span the tibia, fibula, foot, and femur, and the spine example adds CT-RGB-D registration.

Abstract

Registering intraoperative Ultrasound (US) bone-surface point clouds to preoperative Computed Tomography (CT) models can support radiation-free orthopedic navigation. Differences in imaging physics produce distinct surface sampling patterns, complicating cross-modal feature matching. Bone strongly reflects ultrasound and casts an acoustic shadow, so US captures only the probe-facing bone surface and leaves most CT points unmatched. This partial coverage makes correspondences ambiguous on near-symmetric bones. We propose CUFormer, a modality-aware registration framework. We introduce a Modality-Aware Transformer (MAT) that encodes input modality through identity embeddings and updates features through modality-specific adapters within a shared geometric matcher. We model imaging visibility and use predicted reference-side visibility to weight coarse and fine correspondences. CUFormer achieves the lowest mean registration errors among evaluated methods on two CT-US datasets, UltraBones100k and UltraBonesHip, and the CT-RGB-D dataset SpineDepth. On UltraBones100k, it reaches mean rotation error 4.53°, translation error 3.19 mm, and Root Mean Squared Error (RMSE) 2.54 mm, reducing RMSE by 65.8% relative to the best-performing baseline by this metric. These results demonstrate CUFormer's effectiveness in aligning partial bone surfaces with preoperative CT across anatomical sites.

Video

A step-by-step walkthrough on one UltraBones100k test case: the tracked ultrasound sweep and the preoperative CT; which part of the CT the sweep actually sees (the visibility target); how a previous matcher fails on a near-symmetric bone (another test case) and where it places the strip relative to the visible bone; then each CUFormer module (shared encoder and MAT, visibility head, visibility-guided coarse matching, fine matching and pose), located on the main pipeline of the method figure and shown with its real output on this case; finally, one test case each from the fibula, hip and lumbar spine. Intermediate states and poses are saved outputs of the evaluated models; the probe and CT scanner are illustrations, and motions between poses are interpolated for display. The narration is synthesized with the Kokoro text-to-speech model.

Method

CUFormer follows a coarse-to-fine registration pipeline. A shared KPConv-FPN encoder extracts multi-resolution features from the CT reference and the US source. MAT supplies modality identity embeddings and modality-specific residual updates within the shared geometric matcher. Reference-side visibility weights the coarse scores before superpoint correspondence selection and biases the fine scores before Sinkhorn optimal transport establishes point correspondences. A local-to-global estimator then recovers the rigid transform.

Modality-Aware Transformer

Shared geometric features do not explicitly encode the acquisition modality. MAT adds two learnable embeddings, e_ct and e_us, to the coarse features before the transformer and, through a projection φ, to the fine features before matching. The superpoint transformer contains three pairs of geometric self-attention (GeoSA) and cross-attention (CA) blocks. Separate CT and US gated bottleneck adapters follow each shared block, Adapter(x) = x + γ·W_upReLU(W_downx), with γ initialized to zero so that each adapter starts as an identity mapping. The attention layers stay shared, and only the embeddings and adapters are modality specific.

Reference-side visibility

US observes only a thin strip of the bone, so most CT points have no counterpart in the source. CUFormer models imaging visibility through geometric overlap and predicts which CT regions the US observes. The visibility head hierarchically fuses coarse, fine, and full-resolution CT features along the point-to-node hierarchy, and a small per-point network of 1×1 convolutions with group normalization and a sigmoid outputs the visibility probability v_i.

Supervision uses continuous soft targets. For each CT point, the distance d_i to the ground-truth-aligned source is mapped to s_i = clip(1 − d_i/ρ, 0, 1). Points whose target lies in the uncertain band between the negative and positive thresholds are masked.

Visibility-guided matching

At the coarse level, each aggregated visibility becomes a weight w that scales the superpoint scores before global top-K selection, favoring visible CT regions. At the fine level, λ_f log w is added to each reference point's row of point-pair scores before Sinkhorn optimal transport, while the scores of the unmatched (dustbin) entries stay unchanged.

Two-stage training

Stage 1 initializes the shared parameters from GeoTransformer weights pretrained on 3DMatch and trains the matcher, including MAT, with a weighted circle loss on superpoint descriptors and the negative log-likelihood of ground-truth point correspondences. Stage 2 freezes the matcher and trains only the visibility head with a smooth-L1 loss against the soft targets.

Interactive demos

Both demos use held-out test cases and show saved CUFormer predictions. No model runs in your browser.

1 CT on live ultrasound

A real tracked B-mode sweep along a tibia from the UltraBones100k test set. Drag the handle: left of it is the raw ultrasound, right of it is the same frame with the CT cross-section at the CUFormer estimate. The panel on the right shows where the current frame lies on the registered bone.

Raw ultrasoundRaw + CT at CUFormer estimate+ CT

Frame on registered CTFrame

Compare Frame

All 778 frames of the sweep, in acquisition order, placed with the dataset's tracking and calibration. The blue contour is computed from the retained CUFormer estimate for this scan (strict RMSE 1.43 mm); nothing is fitted per frame.

2 Registration explorer

Pick an anatomy and drag the slider: the CT stays fixed while the intraoperative points move from their input pose onto the bone by the CUFormer estimate. Rotate and zoom with the mouse. Colour the CT by predicted visibility to see which part of the bone the model expects the ultrasound to observe.

Input poseCUFormer Predicted visibility Ultrasound

CT reference (blue, fixed) and intraoperative points (orange) from the test sets. The points are moved by the inverse of the estimated CT-to-ultrasound transform, which gives the same relative alignment. Intermediate slider positions interpolate between the two poses for display; CUFormer predicts the final pose in one forward pass.

Results

Comparison on three datasets

We compare CUFormer with ten baselines on the CT-US datasets UltraBones100k and UltraBonesHip and on the additional CT-RGB-D dataset SpineDepth. All methods use three specimen-level cross-validation splits and share the test samples and poses within each split, and every failure stays in the averages.

Method	UltraBones100k CT-US				UltraBonesHip CT-US				SpineDepth CT-RGB-D
Method	RRE ↓	RTE ↓	RMSE ↓	FR ↓	RRE ↓	RTE ↓	RMSE ↓	FR ↓	RRE ↓	RTE ↓	RMSE ↓	FR ↓
ICP	84.16±7.02	64.46±4.62	112.58±14.71	50.8±2.75	97.86±8.19	57.34±1.52	85.36±13.54	53.0±6.94	91.67±11.61	20.02±1.69	36.50±0.64	55.0±10.00
NBReg	41.25±9.30	28.87±3.49	35.21±2.03	27.0±2.75	29.11±13.74	18.84±6.70	19.45±4.97	19.7±9.46	10.97±5.51	2.97±1.32	4.21±1.70	6.7±2.89
PARTE	66.70±9.70	25.21±4.63	40.02±7.84	34.9±7.27	92.35±23.36	47.53±9.03	54.80±10.87	53.0±18.37	128.25±4.79	32.75±2.74	42.84±0.84	81.7±5.77
Predator	16.09±0.45	7.14±0.40	7.42±0.10	1.6±2.75	58.07±20.02	27.30±2.54	32.22±7.03	27.3±12.03	5.65±0.67	3.57±0.22	3.48±0.14	0.0±0.00
CoFiNet	38.99±10.50	16.74±4.64	30.64±20.28	15.9±7.27	77.00±13.91	35.61±9.87	57.50±9.19	37.9±6.94	5.06±0.24	2.98±0.33	3.16±0.22	0.0±0.00
GeoTrans	19.00±4.32	13.19±0.14	13.56±0.39	1.6±2.75	26.04±9.90	16.31±2.00	19.09±4.31	6.1±6.94	21.58±4.37	5.34±1.27	7.82±1.61	11.7±2.89
PANet	94.84±14.52	78.72±6.26	143.23±18.64	55.6±7.27	89.30±3.95	63.69±3.41	90.30±11.67	47.0±6.94	87.05±13.43	35.45±2.03	49.97±2.24	50.0±5.00
MMMI	85.47±7.10	51.74±0.10	87.37±0.98	41.3±5.50	96.94±15.21	45.44±4.42	65.31±2.30	50.0±13.64	118.63±1.62	22.50±0.60	43.24±1.54	85.0±8.66
RORNet	88.19±10.91	46.92±2.45	87.20±18.06	46.0±7.27	89.90±2.82	59.64±4.46	75.21±5.75	48.5±5.25	101.33±19.02	28.59±3.45	38.67±5.00	53.3±20.82
IGASA	51.66±3.48	15.28±1.43	20.36±5.54	22.2±2.75	98.58±4.91	49.80±1.95	46.94±1.28	57.6±6.94	61.46±16.15	10.49±1.52	21.87±4.04	31.7±7.64
CUFormer	4.53±0.83	3.19±0.33	2.54±0.10	0.0±0.00	22.00±2.92	12.55±1.48	15.07±1.41	1.5±2.62	4.72±5.02	2.18±0.93	2.36±1.28	0.0±0.00

Mean ± sample standard deviation across the three specimen-level cross-validation splits. Each split averages over its test scans, including failures. RRE is in degrees, RTE and strict pointwise RMSE are in mm, and FR (%) is the fraction of scans with RRE ≥ 90°. Lower is better, and best means are bold. The upper group contains per-case optimization methods, and the lower group contains learning-based methods, each ordered by publication year. NBReg: NeuralBoneReg. GeoTrans: GeoTransformer.

On the CT-US datasets, CUFormer achieves the lowest mean errors and FR, with 4.53°, 3.19 mm, 2.54 mm, and FR 0.0% on UltraBones100k and 22.00°, 12.55 mm, 15.07 mm, and FR 1.5% on UltraBonesHip. On UltraBonesHip, where the hip anatomy is deeper and harder to image with US, errors increase for the three learning-based baselines with the lowest RMSE on UltraBones100k. CUFormer also degrades there but retains the lowest RMSE. On SpineDepth, CUFormer obtains 4.72°, 2.18 mm, 2.36 mm, and FR 0.0%. Several baselines also perform well on this dataset, and CUFormer still reduces RMSE by 25.3% relative to CoFiNet.

Qualitative comparison

Four CT-US rows show UltraBones100k fibula, tibia, and foot and UltraBonesHip femur, followed by an additional CT-RGB-D SpineDepth L5 row. Baselines are ordered by publication year. GT shows CT in blue and US/depth in orange. Other columns show CT in gray and source points colored by pointwise error on a 0 to 10 mm scale. Red-framed insets enlarge the boxed regions in the fibula and L5 rows.

Component ablation

Setting	RRE ↓	RTE ↓	RMSE ↓
Baseline	19.00	13.19	13.56
+ MAT	7.06	5.90	5.41
+ MAT (w/o pretrain)	15.10	15.46	16.60
+ Visibility	7.95	7.54	7.88
+ MAT + Visibility	4.53	3.19	2.54

UltraBones100k, averaged across three splits. w/o pretrain: shared parameters are not initialized from 3DMatch weights. RRE is in degrees, and RTE and RMSE are in mm.

Qualitative component ablation on UltraBones100k: Baseline, Visibility, MAT, and MAT + Visibility on fibula, tibia and foot cases, with US points colored by displacement from the ground-truth alignment.

Qualitative comparison of the component ablation on UltraBones100k. US points are colored by displacement from the ground-truth alignment in millimeters.

Each module improves Baseline on its own: MAT reduces RMSE from 13.56 to 5.41 mm, and Visibility reduces it to 7.88 mm. Without pretraining, RMSE rises from 5.41 to 16.60 mm, about 3.1× that of MAT, which indicates that initializing the shared parameters from same-modality pretraining is critical when paired CT-US data are limited. On top of MAT, visibility guidance reduces RMSE by 53.0%, from 5.41 to 2.54 mm, compared with 41.9% on Baseline, and combining both modules gives the best results.

Visibility-guided matching

Setting	RRE ↓	RTE ↓	RMSE ↓
No Visibility	7.06	5.90	5.41
+ Visibility Coarse	4.85	3.65	2.85
+ Visibility Fine	4.81	3.60	2.79
+ Visibility Full	4.53	3.19	2.54
+ Visibility GT	4.51	2.85	2.35

Ablation on top of MAT, averaged across three splits. Coarse and Fine fuse predicted visibility at the respective matching stage, Full fuses it at both stages, and GT uses ground-truth visibility as an oracle.

Predicted reference-side visibility on CT surfaces and its deviation from the overlap-derived soft target.

Predicted reference-side visibility. Top: predicted probability v_i. Bottom: deviation |v_i − s_i| from the soft target.

Either stage alone yields most of the gain, and fusing visibility at both stages gives the lowest errors among the predicted-visibility configurations. The remaining gap to the GT-visibility oracle is 0.02°, 0.34 mm, and 0.19 mm, so fusion at both stages recovers 99.2%, 88.9%, and 93.8% of the oracle's improvement in RRE, RTE, and RMSE. On the test scans of one randomly selected UltraBones100k split, the visibility head reaches an Average Precision of 92.04%, an AUROC of 94.81%, an F1 of 86.33%, and an Expected Calibration Error of 0.025.

Robustness to initial misalignment

Mean strict RMSE and mean rotation error versus initial rotation and translation for seven methods on UltraBones100k.

Seven methods on the UltraBones100k test scans of one randomly selected split. The US is rotated about the CT centroid and translated while the CT stays fixed. (a,b) Mean strict RMSE and mean RRE versus rotation at 30 mm translation. (c,d) The same metrics versus translation at 30° rotation.

CUFormer has the lowest mean RMSE at every perturbation level in both sweeps, rising from 2.64 to 3.70 mm between 0° and 90° of rotation, and the lowest mean RRE except at 90°, where GeoTransformer reaches 8.47° and CUFormer 9.29°. The protocol and success and flip rates are given in Supplementary Section S2 of the paper.

Runtime

On NVIDIA RTX 4090 GPUs, the CUFormer forward pass takes 0.055 to 0.115 s per scan on average across the three datasets, while the complete per-case optimization of NBReg takes 13.9 to 23.4 s. The two timing scopes differ: CUFormer excludes CPU neighborhood construction and host-to-device transfer, and NBReg was measured on GPUs shared with other workloads. Details are given in Supplementary Section S1 of the paper.

Citation

@article{tan2026cuformer,
  title   = {{CUFormer}: Modality-Aware {CT}-Ultrasound Point Cloud Registration for Orthopedic Navigation},
  author  = {Tan, Lixing and Hao, Hongye and Gu, Tianrun and Xia, Qing and Guo, Yuting and Li, Shuai and Hao, Aimin},
  journal = {arXiv preprint arXiv:2611.01536},
  year    = {2026}
}