I'm actually implementing my own Pose-Estimation/- and -Refinement pipeline. For this purpose I use one moving mono-camera. Then I take the consecutive images to estimate the pose and triangulate the points (nothing special). In the last step I refine the poses and 3D-points with a bundle adjustment approach.
Generating 3D points with triangulation from consecutive image pairs will give me multiple estimations for one real-world 3D point. In fact, all the estimations refer to the same point. For my understanding, these estimations of the same 3D-points have to be fused in some way. Otherwise the poses were not linked anymore through a common point (see also image below). Further, looking at the equation for the re-projection error in different publications:
turns out, that 3D point (vector a) is only related to j and not to the cameraindex i.
Do I understand that right or do I have to use a different set of 3D points for each camera view? Suppose I've to merge the 3D points, is there any preferable strategy?
Thanks in advance!
Edit: I know, there are already countless implementations for BA. I want to use it for further development...