Proceedings of the Creative Construction e-Conference (2022) 003 Edited by: Miroslaw J. Skibniewski & Miklos Hajdu https://doi.org/10.3311/CCC2022-003
An Alignment Object Detection Method for Automatically Erecting Precast Components
Xiaotian Ye, Ying Zhou, Hongling Guo and Zhubang Luo
Department of Construction Management, Tsinghua University, Beijing, China
Abstract
In the context of intelligent construction, the alignment of precast components is the key to the automated erection of precast components. This can be divided into preliminary and precise alignment. The alignment process depends on alignment objects on site. The preliminary alignment is based on the control surface, while the precise alignment is based on exact control elements. Traditionally, this requires manual operations and communications to determine alignment objects, thus leading to low efficiency and being prone to accidents. To address this issue, taking the typical sleeve-rebar connection during the erection of precast structural components as an example, this research proposes an alignment object detection method based on computer vision. Firstly, an automatic generation algorithm for the hoisting control surface is established by combining image processing and a precast component model database. Then, an automatic extraction and matching algorithm is proposed to match the extracted control elements (i.e., sleeve centers and rebar contours). Finally, two experiments are conducted to verify the feasibility of the alignment object detection method. It is found that the proposed method can extract the basic alignment information for the automated erection of precast components.
© 2022 The Authors. Published by Diamond Congress Ltd.
Peer-review under responsibility of the scientific committee of the Creative Construction Conference 2022.
Keywords: alignment object detection, intelligent construction, precast component alignment.
1. Introduction
Hoisting is one of the most frequent processes on construction sites, particularly for prefabrication housing production (PHP), during which thousands of prefabricated components are transported and erected[1]. In terms of precast concrete components, their alignment is the key to the erection. However, the traditional manual alignment is not only time-consuming but also prone to collision accidents, resulting in tremendous casualties and property losses[2]. That is because drivers have limited environmental perception[3,4], which makes them unable to know the site situation immediately and accurately[1,5]. Existing research focuses on improving the environmental perception ability of crane drivers combined with various sensors to ensure safety[6,7], but neglecting the identification of alignment reference objects, which is very important to the automated erection of precast components.
In practice, different measuring and positioning equipment has been applied to improve the quality and accuracy of precast component alignment. The former enhances the measurement efficiency through total stations, with angle steel locators or positioning brackets assisting the alignment[8], still requiring manual erection and judgment. The latter combines with various positioning devices to measure the real-time distance between alignment objects and lifting components. Zhou[9] carried accurate setting-out to assist in hoisting alignment based on the Global Navigation Satellite System (GNSS), which is still in the experimental stage. Focusing on long-span steel arch bridges, Zhang[10] predicted the trajectory of precast columns to assist in the alignment with deep learning. Wakisaka[11] introduced a comprehensive
construction system, in which some positioning equipment was utilized for alignment, but the system is bulky and costly. The above methods provide some reference for the automatic alignment of precast components, but manual assistance and erection errors still exist, thus also difficult in supporting the automated erection of precast components.
Visualization sensors have become widespread in industrial engineering. Yan[12] developed an automated assembly method with an accuracy of 50μm by converting the pixel coordinates of the assembly position into robot coordinates. Qin[13] proposed an assembly method for large-scale objects based on vision and laser sensors, which can detect polygon edges and achieve alignment elements. Based on the surface roughness and reflectivity, Tang[14] established an optical alignment system to identify and extract key edge features to assemble complex three-dimensional (3D) microstructures automatically. Although the alignment scene in construction is different from those in other industries, the above alignment element extraction methods can still provide a reference for the alignment of precast components.
This research aims to develop an alignment object detection method based on computer vision via taking the typical sleeve-rebar connection during the erection of precast structural components as an example.
The rest of this paper is structured as follows. The method is first presented in Section 2, then two experiments are illustrated in Section 3, and finally a conclusion is drawn in Section 4.
2. Method
As the prerequisite of component erection, the detection of alignment objects not only determines the position where a lifting component should be erected, but also provides relevant site conditions. According to the erection process of precast structural components, the alignment object detection method involves two parts, i.e., the intelligent generation algorithm of hoisting control surface and the intelligent extraction and matching algorithm of alignment elements. The former is for the preliminary alignment, while the latter is for the precise alignment.
2.1. Intelligent generation algorithm of hoisting control surface
In this research, a hoisting control surface is defined as the vertical projection of a precast component on the erection plane when it is correctly placed. As the reference during component erection, the automatic generation of the control surface is the key to realizing the automatic erection of components. Based on the segmented rebar foreground of the trained Faster R-CNN, an intelligent generation algorithm of the control surface is proposed by combining the image processing algorithm and component database. On the one hand, an automatic generation algorithm for the minimum envelope of rebar is established based on the minimum circumscribed polygon. On the other hand, a screening algorithm is proposed to generate the target control surface according to the rebar arrangement in different envelopes.
2.2. Generation of minimum envelopes
Considering the process that the components are lifted to the top of the in-position location and approach the alignment object from far to near, it depends on the prearranged control lines and points that directly affect the alignment accuracy. Therefore, the hoisting control surface can be determined based on the control points and lines. The minimum envelope of rebar is the basis for generating the hoisting control surface, which contains the original control lines, points, and the steel bar arrangement situation. Fig. 1 shows the generation principle of minimum envelopes, i.e., to find the minimum circumscribed polygon of the points in u-v (two-dimensional (2D) pixel) image coordination. The whole generation process is illustrated as follows.
• Step1: A set 𝑃 contains plenty of points 𝑝𝑖(𝑢𝑖, 𝑣𝑖). To find the point with 𝑣𝑚𝑖𝑛 in set 𝑃 (i.e., 𝑝1 in Fig.1.) and add it in a new set 𝑃′;
• Step3: Line 𝑙 continues to rotate clockwise around 𝑝2 until it finds another point in set P (i.e., 𝑝3 in Fig.1.), then add 𝑝3 in set 𝑃′;
• Step4: Determine whether 𝑝2 and 𝑝3 are coincident. If they coincide, the rotating process ends, and the minimum envelop is generated by connecting points in set 𝑃′ in order; Or, 𝑝3 will be added to set 𝑃′ and line 𝑙 will rotate clockwise around 𝑝3. Then, step3 is repeated to obtain points (𝑝4，…，𝑝𝑛) in set 𝑃′ till 𝑝𝑛 coincides with 𝑝1.
Fig. 1. The generation principle of minimum envelopes 2.3. Screening of minimum envelopes
In the early alignment stage, the overhead camera results in an extra-wide field, which means that multiple components are photographed simultaneously with several minimum envelopes. To solve the problem, an automated envelope screening method is presented by comparing the arrangement of rebar (including the envelope area, the number of rebars, the number of columns, the number of rows, and the spacing of rebar) with the components stored in the database. The entire screening process is illustrated as follows.
• Step1: The area of each envelope is calculated and compared with the bottom area of the lifting component to screen out the nonconforming;
• Step2: The number of steel bars in each envelope is calculated and compared with the sleeve number of the lifting component to screen out the nonconforming;
• Step3: The row and column number of steel bars are calculated and compared with those of the sleeves to screen out the nonconforming;
• Step4: The spacing of rebar is calculated and compared with that of the sleeves to screen out the nonconforming;
• Step5: According to the edge information (e.g., the distance between the sleeve and the bottom edge) stored in the database, the screened minimum envelope is further expanded into the hoisting control surface of the lifting component, providing a reference for intelligent hoisting (see Fig. 2).
Fig. 2. The screened minimum envelope and the expanded hoisting control surface
2.4. Intelligent extraction and matching algorithm of alignment elements
According to the generated control surface, the lifting component can approach the target position and stops above the control surface. More detailed information about the alignment elements, such as the sleeve and rebar, is required to realize precise alignment. An intelligent extraction and matching algorithm for precise alignment elements is proposed based on image processing algorithms and technologies.
2.5. Transformation of precise alignment
Precise alignment in this research can be defined as follows: all rebar should be inserted into the corresponding sleeves when the lifting component is erected correctly. According to the theory of solid geometry that three non-collinear points can determine a plane in 3D space, the problem of precise alignment can be transformed into that of determining three non-collinear sleeves in the component.
Considering that the final steel bars pass through the corresponding sleeves, this problem can be further transformed into the alignment of three non-collinear sleeves and steel bars (see Fig. 3).
Fig. 3. Transformation of precise alignment problem
2.6. Alignment element extraction and deviation calculation
system). Based on the segmented sleeve foreground, Canny Detection is also used to extract the discrete contour points of sleeves. In a pixel image, an ellipse can be parameterized as:
𝐴𝑢2+ 𝐵𝑢𝑣 + 𝐶𝑣2+ 𝐷𝑢 + 𝐸𝑣 + 𝐹 = 0 (2)
Therefore, any ellipse can be uniquely determined by the above five parameters. In addition, an ellipse can also be represented by another five geometric parameters, i.e., the center of the ellipse (𝑢𝑐, 𝑣𝑐), the major and minor axis of ellipse (a, b) and the rotation angle ρ of the major axis to the u axis. The above two ellipse representations can be converted to each other, as shown in Equations (3) - (6) [15,16].
𝑢𝑐=𝐵𝐸 − 2𝐶𝐷
4𝐴𝐶 − 𝐵2 (3)
𝑣𝑐=𝐵𝐷 − 2𝐴𝐸
4𝐴𝐶 − 𝐵2 (4)
𝑎 = 2
√
−2𝐹 𝐴 + 𝐶 − √𝐵2+ (𝐴 − 𝐶
𝐹 )
2 (5)
𝑏 = 2
√
−2𝐹 𝐴 + 𝐶 + √𝐵2+ (𝐴 − 𝐶
𝐹 )
2
𝜌 =1
2𝑡𝑎𝑛−1 𝐵
𝐴 − 𝐶 (6)
As the most commonly used method for ellipse fitting, the least square uses the maximum likelihood to achieve an optimal estimate to minimize the square root of the measurement error, assuming that the random error is normally distributed. The least square minimizes the distance between the measurement point and the true ellipse by a set of parameters. 𝑓(𝑢0, 𝑣0) is the algebraic distance from the measurement point (𝑢0, 𝑣0) to the curve 𝑓(u, v) = 0. Assuming that 𝑃𝑖(𝑢𝑖, 𝑣𝑖)(𝑖 = 1,2, … , 𝑛) represents n (n≥5) measurement points on the ellipse contour, the objective function is described as:
𝑓(𝐴, 𝐵, 𝐶, 𝐷, 𝐸) = ∑(𝐴𝑢𝑖2+ 𝐵𝑢𝑖𝑣𝑖+ 𝐶𝑣𝑖2+ 𝐷𝑢𝑖+ 𝐸𝑣𝑖+ 𝐹)2
𝑛
𝑖=1
(7)
Each coefficient is determined by minimizing the objective function. According to the extreme principle, the first derivative of zero is a necessary but insufficient condition for the extreme value point. Therefore, if the value of the objective function 𝑓(A, B, C, D, E)) is minimized, there must be:
𝜕𝑓
𝜕𝐵=𝜕𝑓
𝜕𝐶=𝜕𝑓
𝜕𝐷=𝜕𝑓
𝜕𝐸=𝜕𝑓
𝜕𝐹= 0 (8)
A system of linear equations can be obtained, and the values of the equation coefficients A, B, C, D, E, and F can be obtained after solving the equations. Fig. 4 shows the ellipse fitted by the least square, with the ellipse center further extracted. Although the ellipse center (𝑢𝑐, 𝑣𝑐) has been elaborated in Equations (3) and (4), its 3D coordinates cannot be directly obtained. This is because the ellipse center point in 2D image coordinates corresponds to the inner wall of the sleeve in the point cloud data (the center point Q of the ellipse is shown in Fig. 4). Therefore, this research calculates the image coordinates of the ellipse's major and minor axis vertices based on the above equations.
(a) Original picture of sleeves (b) Ellipse fitting results Fig. 4. The fitting ellipses for sleeves
The image coordinates of the four vertices of the major and minor axes of the ellipse are described as follows:
(𝑢𝑐+ 𝑎 ∗ 𝑐𝑜𝑠 𝜌 , 𝑣𝑐+ 𝑎 ∗ 𝑠𝑖𝑛 𝜌) (9)
(𝑢𝑐− 𝑎 ∗ 𝑐𝑜𝑠 𝜌 , 𝑣𝑐− 𝑎 ∗ 𝑠𝑖𝑛 𝜌) (10)
(𝑢𝑐− 𝑏 ∗ 𝑠𝑖𝑛 𝜌 , 𝑣𝑐+ 𝑏 ∗ 𝑐𝑜𝑠 𝜌) (11)
(𝑢𝑐+ 𝑏 ∗ 𝑠𝑖𝑛 𝜌 , 𝑣𝑐− 𝑏 ∗ 𝑐𝑜𝑠 𝜌) (12)
Then, the 3D coordinates (𝑥0, 𝑦0, 𝑧0) of the sleeve center can be obtained by taking the average of the four endpoints. After extracting the alignment elements, the deviation between the sleeve center and the fitting line of the rebar can be calculated, which is exactly the distance d from Point Q to the corresponding Line L. The distance d can be calculated according to Heron's formula. When d is less than the value required by the standard, the sleeve and the steel bar can be aligned.
3. Experiment and test
Two experiments were designed to validate the feasibility of the proposed method. Experiment 1 was for the generation of minimum envelopes, and experiment 2 was for the extraction and matching of precise alignment elements. In experiment 1, the minimum envelopes of rebar were generated based on the segmented rebar foreground, as shown in Fig. 5. Rebar_area2 was selected by comparing the arrangement of rebars (including the envelope area, the number of rebars, the number of columns, the number of rows, and the spacing of rebar) with the components stored in the database.
In experiment 2, two 3D models represented the rebar and sleeves, respectively, and the accurate elements were identified and extracted (see Fig. 6). From left to right, the extracted lines are 𝑙12, 𝑙13 and 𝑙14, and the corresponding centers are 𝑄12, 𝑄13 and 𝑄14, whose pixel coordinates and 3D coordinates are shown in Table 1. Each extracted line was calculated based on two random points on them. Finally, the deviation distance between 𝑄𝑖 and 𝑙𝑖 (𝑖=12, 13, 14) was calculated based on Heron's formula, whose results are shown in Table 2. The average error of the offset is 0.003m, which is acceptable.
(a) 3D Model (b) Alignment element extraction Fig. 6. Sleeve-rebar alignment element extraction based on 3D models Table 1. Extraction results of sleeve-rebar alignment elements
Alignment
elements 𝑢 𝑣 𝑥 𝑦 𝑧
𝑄12 400 409 0.0890469 -0.0157684 0.338143
Two points on 𝑙12
396 328 0.0883307 -0.0400191 0.33
390 159 0.287529 -0.0142995 0.548636
𝑄13 536 536 0.0441957 0.0670145 0.380333
Two points on 𝑙13
538 398 0.0325581 -0.00991643 0.2855
536 296 0.0317233 -0.0236569 0.273
𝑄14 696 696 -0.0197086 0.0670236 0.346364
Two points on 𝑙14
680 388 -0.0135481 -0.00818422 0.3422
684 368 -0.01442 -0.0540234 0.328333
Table 2 Sleeve-rebar alignment deviation calculation
𝑄12 and 𝑙12 𝑄13 and 𝑙13 𝑄14 and 𝑙14
Deviation calculation 0.024 0.019 0.019
Actual deviation 0.029 0.019 0.015
Absolute value of error 0.005 0.000 0.004
Average absolute value of
error 0.003
4. Conclusion
To enable the automatic erection of precast components, this research proposes an alignment object detection method. An intelligent generation algorithm of the hoisting control surface is developed to realize the preliminary alignment by extracting and screening the minimum envelope of rebar, while an intelligent extraction and matching algorithm is proposed to achieve precise alignment by extracting sleeve-rebar alignment elements and calculating their deviation distance. It is shown from experiments that the proposed method is feasible and efficient. However, the tests are conducted in a laboratory environment, and it is assumed that the rebars are not bent. Thus, the method can be verified in real construction scenarios in the future.
References
[1] Z. Lei, M. Al-Hussein, U. Hermann, et al., „Heavy lift analysis at FEED stage for industrial project,” 2016 Winter Simulation Conference (WSC), pp. 3281-3289, 2016. https://doi.org/10.1109/WSC.2016.7822359.
[2] T. Cheng, J. Teizer, „Modeling tower crane operator visibility to minimize the risk of limited situational awareness,” Journal of Computing in Civil Engineering, tom 28, nr 3, pp. 04014004.1-04014004.15, 2014. https://doi.org/10.1061/(ASCE)CP.1943- 5487.0000282.
[3] Y. Fang, Y.K. Cho, „Effectiveness analysis from a cognitive perspective for a real-time safety assistance system for mobile crane lifting operations,” Journal of Construction Engineering & Management, tom 143, nr 4, pp. 05016025, 2016.
https://doi.org/10.1061/(ASCE)CO.1943-7862.0001258.
[4] A. Shapira, Y. Rosenfeld, I. Mizrahi, „Vision system for tower cranes,” Journal of Construction Engineering & Management, tom 134, nr 5, pp. 320-332, 2008. https://doi.org/10.1061/(ASCE)0733-9364(2008)134:5(320).
[5] A. Shapira, Y. Rosenfeld, „Achieving construction innovation through academia-industry cooperation—keys to success,” Journal of Professional Issues in Engineering Education & Practice, tom 137, nr 4, pp. 223-231, 2011.
https://doi.org/10.1061/(ASCE)EI.1943-5541.0000057.
[6] Y.C. Chen, H.L. Chi, S.C. Kang, et al., „Attention-based user interface design for a tele-operated crane,” Journal of Computing in Civil Engineering, tom 30, nr 3, pp.04015030.1-04015030.12, 2016. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000489.
[7] Y. Fang, Y.K. Cho, „Crane load positioning and sway monitoring using an inertial measurement unit,” Computing in Civil Engineering 2015, pp. 700-707, 2015. https://doi.org/10.1061/9780784479247.087.
[8] L.G. Shen, H.M.Wei, Y.H. Liang, et al., „Precise positioning technology of pc column hoisting in prefabricated building,”
STRUCTURE CONSTRUCTION, tom 42, nr 9, pp. 1671-1673, 2020. https://doi.org/10.14144/j.cnki.jzsg.2020.09.024.
[9] M.D. Zhou, D. Luo, K.L. Ding, et al., „GNSS accurate positioning-point and setting -out of hoisting operation for construction tower crane,” Bulletin of Surveying and Mapping, tom 11, pp. 60-63, 2019. https://doi.org/10.13474/j.cnki.11-2246.2019.0352.
[10] K. Zhang, S. Tong, H. Shi, „Trajectory prediction of assembly alignment of columnar precast concrete members with deep learning,” Symmetry, tom 11, nr 5, pp, 629.1-629.20, 2019. https://doi.org/10.3390/sym11050629.
[11] T. Wakisaka, N. Furuya, Y. Inoue, et al., „Automated construction system for high-rise reinforced concrete buildings,”
Automation in Construction, 2000, tom 9, nr 3, pp. 229-250, 2000. https://doi.org/10.1016/S0926-5805(99)00039-4.
[12] J.J. Yan, H.W. Wang, G.J. Yu, et al., „Automatic precision assembly technology of satellite antenna module based on machine vision,” Measurement & Control Technology, pp. 1-10, 2021. https://doi.org/ 10.19708/j.ckjs.2021.05.235.
[13] Z.Qin, P.Wang, J.Sun, et al., „Precise robotic assembly for large-scale objects based on automatic guidance and alignment,” IEEE Transactions on Instrumentation and Measurement, tom 65, nr 6, pp. 1398-1411, 2016.
https://doi.org/10.1109/TIM.2016.2526738.
[14] Y.L. Tang, Z.J. Zhang, X. Ye, et al., „Micro-assembly precise coaxial alignment methodology based on surface roughness and reflectiveness matching,” Assembly Automation, tom 34, nr 2, pp. 141-150, 2014. https://doi.org/10.1108/AA-03-2013-029.
[15] E.S. Maini, „Enhanced direct least square fitting of ellipses,” International Journal of Pattern Recognition and Artificial Intelligence, tom 20, nr 06, pp. 939-953, 2006. https://doi.org/10.1142/S021800140600506X.
[16] A. Fitzgibbon, M.Pilu, R.B. Fisher, „Direct least square fitting of ellipses,” IEEE Transactions on pattern analysis and machine intelligence,” tom 21, nr 5, pp. 476-480, 1999. https://doi.org/10.1109/WSC.2016.7822359.
