You are on page 1of 5

Multi-view point clouds registration and stitching based on SIFT feature

Jun Chu
Institute of Computer Vision, Nanchang Hang kong University
Nanchang, Jiangxi 330063, China

Chun-mei Nie
Institute of Computer Vision, Nanchang Hang kong University
Nanchang, Jiangxi 330063, China

e-mail: chujun99602@163.com

e-mail: niechunmei83@126.com

Abstract: In order to solve multi-view point clouds registration in large non-feature marked scenes, a new registration and stitching method was proposed based on 2D SIFT(Scaleinvariant feature transform) features. Firstly, we used texture mapping method to generate 2D effective texture image, and then extracted and match SIFT features, obtained accurate key points and registration relationship between the effective texture images. Secondly, we reflected the SIFT key points and registration relationship to the 3D point clouds data, obtained key points and registration relationship of multi-view point clouds, we can achieve multi-view point clouds stitching. Our algorithm used texture mapping method to generate 2D effective texture image, it can eliminate interference of the holes and ineffective points, and can eliminate unnecessary mistake matching. Our algorithm used correct extracted matching point pairs to stitch, avoiding stepwise iterated of ICP algorithm, so our algorithm is simple to calculate, and its matching precision and matching efficiency are improved to some extent. We carried experiments on two-view point clouds in two large indoor; the experiment results verified the validity of our algorithm. Keywords: Multi-view point clouds registration; multi-view point clouds stitching; SIFT key point; texture mapping

I.

INTRODUCTION

Because it is widely used in mobile robot navigation, urban planning and protection of cultural relics, the large scene modeling technology based on image becomes an important research direction in the area of computer vision and computer graphics. Be restricted by the view angle of camera, single viewpoint image can only obtain local 3D information of local scene, so how to registration and stitching point clouds of different viewpoint becomes a problem to be solved in large scene modeling. ICP[1] algorithm is the classical algorithm of the point clouds registration and stitching. It derives many algorithms on that foundation [2, 3, 4], these algorithms established mapping relationship of corresponding point pairs according to implicit geometric constraint of point clouds data. These algorithms have a large amount of calculation, mainly to resolve multi-view point clouds stitching problem of single object. Large scale scenes has a mass of 3D point data, so it is very difficult to extract geometric features, especially when the scenes have occlusion or constant curvature of object surface in the scenes. Feature extraction of image has
___________________________________ 978-1-61284-840-2/11/$26.00 2011 IEEE

achieved well, scholars have put forward many algorithms of feature detection and extraction which is robustness to illumination change, partial occlusion and scale variation, such as Davids[7]SIFT descriptors. Therefore, combining 2D image with 3D point clouds to complete multi-view point clouds registration can improve velocity and accuracy of the stitching. Bendels[8] put forward a algorithm of multiview registration based on image feature surface elements. This method extracted SIFT feature point from 2D image, and estimated the correspondence 3D feature area according to the scale-space of SIFT method , then used ICP iteration to register and stitch point clouds in the selected 3D feature area. Although this algorithm was based on image feature and used image features to register and stitch, which has decreased computational complexity compared with ICP algorithm, it also doesn't need to add special markers in reconstruction scenes, but it still needs ICP iterating calculation, and cant eliminate interference of holes and invalid 3D point clouds. This paper, based on Bendels[8] work, put forward a new algorithm of multi-view registration and stitching. Firstly, we extracted SIFT features and match them in 2D effective texture image, then mapped the extracted SIFT features and matching relationship into the 3D point clouds data and obtain features and matching relationship of multiview 3D point clouds, finally achieved multi-view point clouds stitching. II. GENERATION TEXTURE IMAGE OF EFFECTIVE POINT
CLOUDS DATA

Texture image of effective point clouds data is corresponding to 2D image of every viewpoint effective 3D point clouds data. We can use scanner or camera to obtain 3D point clouds, but no matter what method we choose, which it cant avoid holes and noise, in order to decrease the effect of registration and stitching precision about multiview point clouds, we use mutual mapping method between 3D point clouds and 2D texture image to obtain texture image of effective point clouds data. Texture mapping is mainly to solve mapping relationship between texture space and scenery space.In our system, single view point clouds and texture image were obtained at the same time, so we have clear mapping relationship between texture space and scenery space.We

need to establish mapping relationship between effective texture space and effective point clouds data. Firstly, we projected the 3D point clouds to 2D plane, secondly, used projection binary picture to make 8 neighborhood seed filling and area filling, so that we can obtain projection graph of effective point clouds data. We assume some points are known in hypothesis district area, we call them as seed points, then searched adjacent points of the seed point, if this point found by us meets the filling conditions and not be visited, then filled this point and regarded the point as a new seed point, repeat the operation until all the points had been visited. Definition: use function F ( x, y ) to describe seed filling algorithm, inputting seed point, and outputting matrix that had been filled, define symbol [1] x, y to represent matrix which had been filled with point(x,y), symbol [0] x, y to represent matrix which had not been filled with point(x,y), so we used recursive function to express seed filling algorithm, as follows:
1 1 [1]x, y + F(x +1, y + i) + F(x 1, y + i) + = F(x, y) = F(x +1,iy1+ F(x 1, y), i=1 C(x, y) = true ) ; ; [0]x, y , C(x, y) = false

of 3D point clouds is based on stereo vision, only the middle area is the overlap area of stereo image pair. We can obtain the 3D point clouds from that area. The area which was marked by pane, as shown in Fig 1(b), is hole area, it corresponds to holes of the 3D point clouds. Because the area can result in mistake matching or can not find match in the 3D reconstruction which is based on stereo vision, so that we cant obtain 3D point clouds in this area. This method effectively eliminated interference of invalid data and holes, and make 3D effective point clouds data and 2D image pixel corresponding to each other, so our method can effectively reduce error matching in subsequent match. III. 1) EXTRACTION AND MATCHING WITH SIFT FEATURES AND 3D FEATURES

(1)

Where C is filling condition, when C(x,y)=true, it means that 2D mapping point (x,y) of effective point clouds satisfies the filling conditions, while C(x,y)=false, it means that 2D mapping point (x,y) of invalid point clouds does not satisfy the filling conditions, C is the 2D point which the 3D point clouds mapping into the plane. Assume F is the filled 2D projection of the effective point clouds data, texture image which is corresponding to point clouds is denoted by I , so texture image of effective point clouds data can be denoted by as following: M = F *I (2)

Extraction and matching with SIFT features SIFT is a local feature which is proposed by David[7]. By searching extreme value point through scale space, we can extract the SIFT feature which is under translation, scale and rotation.This paper used SIFT algorithm to extract 2D features, then used RANSAC[9] to eliminate error matching, the steps are as follows: (1)Feature point extraction: using SIFT algorithm to extract feature point in every effective texture image; (2)Assumption corresponding: calculate feature points matching set according to their gray similar to the neighborhood of the feature point, (3)RANSAC robust estimation: sample M times, where M is determined by adaptive method of RANSAC; (4)Select four groups corresponding points to compose a random sample, and calculate the number of the inliers corresponding to H ; (5)Calculate distance d for every assumption corresponding group; (6)Determine the number of the correspondence points, and calculate the number of interior point which corresponds to H, In accordance with
d < t ,where t = 5.99 ( is standard deviation of

(a) Original 2D texture image

(b) Texture image of effective point clouds data

Figure 1. Texture image of effective point clouds data of a certain scene

pixel point in neighborhood area); (7)Select H which has maximum interior point, if there are more than one H which has equal interior points, we select the one H which has the minimum standard deviation of interior point. 2) 3D feature point extraction Pixel point of effective texture image, which is obtained by point clouds texture mapping has one-to-one correspondence relationship to 3D point clouds[10].

Fig.1(a) is original 2D texture image; Fig 1(b) is texture image of effective point clouds data. Obviously, Fig 1(b) has less scene area than Fig 1(a), because the obtaining

p=

1 n

p
i =1

q=

1 n

q
i =1

(4)

the matching points pairs are as follows:


pi' = pi p

qi' = qi q

(5)

error equation of matching point after standardization is as follow:

Figure 2. The relationship between 2D feature points and 3D point clouds feature

As shown in Fig.2, because the viewpoint of camera, the point U j of image and Pj of 3D scene are on the same straight line, so at first we extracted feature point of effective texture image, then used reflection method to find corresponding point in 3D point clouds, which corresponds to 2D feature points, the methods are as follows: (1) We used SIFT algorithm to extract feature points in 2D texture image, then calculated the coordinate of the point. Because more than three matching pairs can stitch the scene, in this paper we select four matching pairs points of 2D image, which can decrease computational complexity; (2) Because 2D feature points and 3D point clouds have one-to-one correspondence relationship, we can calculate coordinate of corresponding feature points of 3D point clouds. IV. 1) 3D POINT CLOUDS STITCHING

(6) Substitute the equation (4) and equation (5) into equation (6), we can obtain:

= q Rp
2 ' i i =1 n 't ' i i 't i

' i

(7) The equation (7) indicates that if we want to get it's minimum we must make sure that the equation (8) get the maximum

= (q q + p
2 i =1 n

pi' 2qi' Rpi'

q Rp
't i i =1

' i

(8)

So:

q
i =1

't i

n t Rpi' = Trace qi' Rpi' i =1

(9)

Suppose H =

q
i =1

't i

pi' , and apply the SVD to H , we

coarse stitching of point clouds The Multi-view 3D point clouds stitching is to make coordinate transformation of point clouds in different coordinate systems, the main problem is to estimate the coordinate transformation parameters R (rotation matrix) and T (translation sector). According to the matching point pairs which are obtained through the above step, we can estimate coordinate transformation relationship, that is to estimate parameters R and T , which make the objective function minimum:
E=

can obtain U and V , so when R = VUT , the equation (3) has the minimum value, the translation matrix is: (10) T = q Rp 2) Accurate stitching Due to the existence of calculation error,so the different multi-view point clouds which have been made coordinate transformation to a unified coordinate system cant coincide with each other very well. We project pi qi of 3D characteristics matching pairs onto YOZ plane after coordinate transformation. We use lines to connect pi qi with origin of coordinates respectively,assume pi is the angle between opi and X axis, is:

q (Rp + T )
i i i =1

(3)

qi

is the angle between

oqi and X axis,so the average angle between opi and oqi
i = pi qi
(11)

Shen hai ping[11] put forward an algorithm which is based on least square method to estimate coordinate fitting matrix between different view point clouds, this algorithm resulted in the data malformation, so we used SVD method which was proposed by K.S. Arun[12] to estimate rotation matrix R and translation matrix T , it can elimilate noise interference to some extent. Assuming the { pi } and {qi } are matching points pairs of 3D point clouds in two consecutive viewpoint, their gravity center are denoted by p and q respectively, they satisfy the following equation:

the average angle between of projections of all 3D matching point pairs which project onto YOZ plane is:

(12) n The other average angles between of projections of all 3D matching point pairs which project onto XOZ XOY plane are:

(
i =1

pi

qi )

(
i =1

pi

qi )

(13)
qi )

(14) n Because of the existence of calculation error,the projections of 3D matching point pairs, which project onto three projection planes XOY XOZ and YOZ are not

(
i =1

pi

completely coincide with each other in unified coordinate system which has already been transformed, their error angle are , , .Point set { pi } need to interwine X axis to rotate angle , interwine Y axis to rotate angle , interwine Z axis to rotate angle , that in order to make it close to point set {qi } .so the rotation matrix is Rxyz = Rz Ry Rx ,after calculating:

Rxyz

cos cos cos sin + cos sin sin sin sin + cos sin cos = sin cos cos cos + sin sin sin cos sin + sin sin cos sin cos sin cos cos
V. EXPERIMENTAL RESULTS AND ANALYSIS

(15)

Translate the 3D points again , which means to project the same name matching points onto X axis Y axis and Z axis, the difference of average vector are

ox pi oxqi n

oy pi oyqi n
point clouds is:

oz pi ozqi n

,the average translation of 3D

ox oxqi T ' = pi n

oy pi oyqi n

oz pi ozqi n

(16)

In order to verify the efficiency of our algorithm in this paper, we carried out an experiment on two-view point clouds which are extracted from indoor scene. Fig.(3) are the stitching results of indoor scenes. Fig 3(a) and Fig 3(b) are the SIFT features of the two adjacent frames of effective texture image in indoor scenes, the match feature points which are selected by random sampling are marked by red pane,the blue arrows represent SIFT feature points of effective texture image; Fig 3(c) and Fig 3(d) are 3D feature of point clouds data which correspond to effective texture image, the red * markers are 3D feature points which are extracted by our algorithm, Fig 3(e), Fig 3(f) and Fig 3(g) are experimental results of indoor scenes which are reference[8] and obtained by using our algorithm reference[1] respectively.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

Fig.3(a) is feature extraction of the 7th frame effective texture image, Fig.3(b) is feature extraction of the 8th frame effective texture image, Fig.3(c) is feature extraction of point clouds which correspond to the 7th frame effective texture image, Fig.3(d) is feature extraction of point clouds which correspond to the 8th frame effective texture image, Fig.3(e) is point clouds stitching of our algorithm, Fig.3(f) is point clouds stitching of reference [8], Fig.3(g) is point clouds stitching of reference [1]
Figure 3. experimental results of indoor scene

ICP method used all overlap region of 3D point clouds to match, because the scene exists dense point clouds, the computation is complex, the time of scene registration is more than one hour, the mean square deviation of indoor scene is 3.18mm. Bendelss method[8] considered the 3D point clouds which corresponding to scale-space size of 2D feature as 3D stitching feature, this method located part of feature points pairs and reduced the complexity of computation, the time of scene stitching is increased to 563.5 seconds, the mean square deviation of indoor scene is 2.45mm. Our algorithm used texture mapping to generate effective texture image of 2D, it eliminated interference of invalid point clouds data and holes, and avoided unnecessary error matching, in addition that our algorithm used correct matching point pairs to stitch, it avoided stepwise iteration of ICP algorithm, so it is simple to calculate, and matching precision and matching efficiency are improved greatly. Matching time and matching rate of these three methods are listed in the table 1, stitching time and stitching error of these three methods are listed in the table 2.
TABLE Matchi ng matching time and matching rate of these three methods
Experimental method matching accuracy matching time (s)

that effective 3D feature which corresponding to every 2D feature can be found in the 3D point clouds, it can eliminate the unnecessary error matching, so matching efficiency and matching precision were improved. Our algorithm used the correct matching point pairs to stitch, so it can avoid stepwise iterative of ICP algorithm, and decreased computational complexity of matching, it can also reduce stitching error which is brought by error matching. The theoretical analysis and experiment results showed that our algorithm is simple to implement, and has better matching effect and higher precision. it is suitable for constructing large and complex indoor and outdoor three-dimensional topography. Unfortunately, our method can only achieve registration and stitching of two view-point data in large scene. In the future , we will study registration and stitching of multi-view data, the main aim is to solve error accumulation problem of multi-view stitching. REFERENCES
[1] P. J. Besl , N. D. McKay, A Method for Registration of 3-D Shapes, IEEE Trans. Pattern Analysis and Machine Intelligence, 1992, 18(5): 239-256. [2] Bae K H , Lichti D D. A method for automated regist ration of unorganized point clouds [J] . Journal of Photogrammetry and Remote Sensing , 2008 , 63 (1) :36-54. [3] Liu Y H. Improving ICP with easy implementation for free form surface matching [J] . Pattern Recognition , 2004 , 37 (2) : 211-226. [4] Liu Y H. Improving ICP with easy implementation for free form surface matching [J] . Pattern Recognition , 2004 , 37 (2) : 211-226. [5] LuoXianbo, Data registration in 3-D scanning systems[J], Tsinghua Univ (Sci & Tech),2004 44(8):1104-1106. [6] Xie Zexiao, A Fine Registration Method for 3D Point Clouds in Reverse Engineering,,mechanical engineering, 2009,20(13):15771580. [7] David G. Lowe, Distinctive Image Features from ScaleInvariant,International Journal of Computer Vision, 2004,60(2): 1573-1405. [8] Gerhard Heinrich Bendels, Image-Based Registration of 3D-Range Data Using Feature Surface Elements.In Proceeding of the 5th international symposium on Virtual Reality, Archaeology and cultural Heritage, Eurograohics,2004, 115-124. [9] M.A.Fischler, R.C.Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Application to Image Analysis and Automated Cartography[J]. CACM, 1981, 24(6):381~395. [10] Eric R. Smith, Bradford J. King, Charles V. Stewart, Richard J. Radke, "Registration of Combined Range-Intensity Scans: Initialization through Verification", Computer Vision and Image Understanding, 2007,110(2): 226244. [11] Shen Hai-ping,, Research of Point-clouds Registration Based on Lea st-square Method, Journal of Image and Figics,2005,10(9): 11121116. [12] K.S.Arun, Least-Squares Fitting of Two 3-D Point Sets, IEEE Trans. Pattern Analysis and Machine Intelligence,1987,9(5):698-700.

Reference 1 method Reference 8 method our algorithm

80% 90% 96%

130 60 34

TABLE Stitching time and stitching error of these three methods Stitchi ng
Method (mm) time (S) mean square deviation (mm)

Reference 1 method Reference 8 method our algorithm

5800 563.5 96

3.18 2.45 1.35

VI.

CONCLUSION

Our algorithm used SIFT feature of effective texture image to achieve registration and stitching of dense multiview point clouds, we obtained texture image of effective point clouds through mutual mapping between 3D point clouds and 2D texture image, this algorithm eliminate interference of holes and invalid point clouds, and ensure