原路径：/SSD2/yangzijing/egoview-v9-foundationhand-s9l10-20260901/docs/V9_MODEL_REPLACEMENT.md
SHA256：f6030a03457a818dfe0437684908fa40192c2acdb387eb45827396b84d35f10d
1 # V9: FoundationHand as the only 3D hand source 2 3 `python -m v9_fusion.foundationhand_pipeline` renders a completed V9 sequence 4 with the pinned `FoundationHand-S9L10 (1e-6)` replacing the fused hand provider. 5 This is a cached pipeline replay. It does not run HaWoR reconstruction, the 6 HaWoR-based Stage 3/4 optimizer, or the whole-block V7 geometry fallback. 7 The ordinary `python -m v9_fusion` entry point still uses fusion. 8 9 The replay retains the existing V9 temporal left/right association, YOLO 10 dropout classification, conditional SAM3 confirmation, solid MANO renderer and 11 dense stereo scene. The previous completed scene and 12 camera trajectory are held fixed, including that scene's existing exclusion 13 mask. DA3 is not invoked. The FoundationHand inference is reused with its 14 checkpoint, prediction and input bindings checked, including the original 15 T=8, stride=4 configuration. 16 17 The default `--hand-estimation calibrated-sequence` repairs the calibrated 18 stereo route. Geometry and display both use the rectified sensor K (about 19 688 px focal length for the supplied camera) rather than projecting native 20 S9 geometry with the legacy canonical K (960 px). The generic cached 21 NormalizedInput still describes the inherited RGB raster contract; the final 22 hand's `intrinsics` and explicit calibrated camera metadata are authoritative 23 for geometry and the renderer verifies that it uses the same K. 24 25 `v9_fusion.s9_sequence_fit` runs full-sequence MANO estimation from S9's 26 rotations, shape and wrist using the pinned official MANO decoder. Only 27 MMPose 2D observations, confidence, detector boxes and qualified presence 28 enter from V9. Subject shape is shared over the sequence; wrist Z is fixed 29 to S9's prediction. Pose and root XY are estimated jointly with RGB evidence, 30 the S9 pose prior and temporal continuity. All temporal variables belong to 31 one sequence, so there are no independent optimization-window boundaries. 32 Large palm reversals are initialized from calibrated RGB wrist/MCP observations 33 using rigid-palm PnP proposals. Only proposals outside the original 0.7-radian 34 local basin and improving RGB residual by at least 25% replace the global 35 orientation seed. This leaves the source finger rotations and wrist Z intact. 36 The optimizer then allows large whole-hand turns while retaining the 0.7-radian 37 per-joint finger trust region. A separate temporal term in palm coordinates 38 prevents finger articulation from compensating for abrupt global orientation 39 changes. This fixes the folded fingers observed during palm turns; there is no 40 video ID or timestamp-dependent repair. The default uses 360 optimization steps. 41 This is test-time **geometric optimization**, explicitly recorded as such, 42 and not raw network output. There is still only one learned 3D model. 43 44 The calibrated route disables the old renderer-only smoothed translation. 45 No averaging or smoothing is applied to the final mesh after estimation. 46 Joint geometry feeds camera labels, world labels and both display panels. 47 Full sequence estimation takes about eight seconds on a 4090 for this video; 48 foundation inference and earlier RGB/stereo caches are reused independently. 49 50 Only an explicit allowlist of boxes, detector flags, MMPose keypoints/scores, 51 frame identity and camera metadata enters the new hand from the old cache. 52 Old joints, vertices, pose, shape, final valid mask and geometry-dependent 53 quality decisions are excluded. An unknown quality tier is encoded as zero 54 for the existing display policy. Cached 2D observations were originally 55 produced inside V9's shared model run; cache reuse is recorded separately from 56 new model invocations. 57 58 Presence can also be confirmed by independent RGB evidence: a direct YOLO 59 score of at least 0.6 and at least eight usable MMPose joints scoring at least 60 0.6, inside the image and the expanded detector box. Spatially ambiguous 61 duplicate confirmations are rejected. This restores the RGB quality role 62 without borrowing a HaWoR geometry-quality flag. Existing duplicate and 63 outward-exit gates still run. On the supplied video it restores the 12 64 re-entry hand frames missed by the first pure-model replay. 65 66 SAM3 runs only for bounded fast-motion internal gaps after the same-slot YOLO 67 association. Outward exits cannot be restored by SAM3. A sequence with no 68 eligible gap correctly records zero SAM3 model invocations; the controller 69 still executes and supplies masks to final presence qualification. The 70 existing conservative handling of bounded nonfast gaps is retained. 71 72 "Only FoundationHand" describes the learned 3D geometry source and the 73 initialization/prior of the geometric estimator. The rejected earlier 74 behavior remains available as `--hand-estimation legacy-adapt` for controlled 75 comparison; that mode keeps canonical camera-XY and display-offset policies. 76 The default uses the calibrated estimator above. The final translation field 77 follows V9's joint-0 wrist contract, 78 not a native MANO transl parameter. Native MANO hand-size metadata is not 79 available in this replay and is encoded as NaN rather than copied from V7. 80 81 Example for the requested recording: 82 83 ```bash 84 CUDA_VISIBLE_DEVICES=2 PYOPENGL_PLATFORM=egl OMP_NUM_THREADS=8 OPENBLAS_NUM_THREADS=8 \ 85 /home/yangzijing/miniconda3/envs/egoview/bin/python -m v9_fusion.foundationhand_pipeline \ 86 --baseline /SSD2/yangzijing/v9_stereo_demo_runs_20260908/20260907-171916_v9_source_fixed \ 87 --prepared artifacts/20260907-171916_v9_prepared_full_20260907/prepared_manifest.json \ 88 --v9-state-cache /SSD2/yangzijing/v9_stereo_demo_work_20260908/20260907-171916_v9_source_fixed/v7_prediction_state_cache \ 89 --output /path/to/new/output --device cuda:0 90 ``` 91 92 The output contains the video, its renderer receipt, the shared hand/world 93 bundle, the YOLO/SAM3 controller artifacts and a run manifest with model source 94 counts. This bundle has its own replay contract; it does not claim the old 95 HaWoR quality certification or training eligibility. It is the requested 96 pure-model video branch, with a calibrated repair of its geometry estimation. 97 98 Focused integration tests verify that perturbing the old hand geometry and 99 quality masks cannot change the replacement output, SAM3 confirmation changes 100 the final fast-gap validity, unbounded missing hands are suppressed despite 101 finite network outputs, and mismatched frame identities are rejected. 102 Additional tests check calibrated ray initialization, independent RGB presence, 103 fixed wrist Z, coherent fitted geometry publication and preservation of masks. 104 Palm tests cover a large reversed orientation, proper rotations without local 105 finger deformation, retention of small-angle seeds, missing observations and 106 edge-on collinear palms. The previous sequence estimator can be reproduced in 107 its worker with `--no-palm-initialization --global-rotation-limit .7 108 --articulation-temporal-weight 0 --steps 180` for controlled comparisons. 109 110 Validation uses the same visible hand frames and RGB keypoints across versions. 111 Every fifth frame is withheld in a separate verification run, including from 112 root XY initialization. Held-out RGB reprojection improves as well. These 113 target-free measures are not ground-truth MPJPE, and second differences include 114 real motion. The repair does not claim a dataset benchmark improvement.