4 CONCLUSIONS
2.4 A G RADIENT -B ASED M ETHOD : S IMULTANEOUS E STIMATION AND SEGMENTATION OF THE O PTICAL
When estimated and segmented motion information is not satisfactory due to heavy noise or other effects, then we can run the motion estimation and segmentation in one optimization process. In [40] and [65] this technique is used with the correlation approach. Since motion information should be reconsidered many times during the segmentation process the above algorithms can be extremely time consuming.
On the contrary, we have chosen the gradient approach for optimization, because the continuous reevaluation of the SSD in the correlation-based model does not fit our parallel framework. In other words, the spiral movement of frames made it possible to calculate the SSD in all pixel locations simultaneously but an optimization involving the motion model would require the repetition of the spiral image translation process.
This problem is resolvable with using the gradient-based estimation approach, where the evaluation of a possible motion vector candidate can be achieved for all pixels simultaneously in the parallel framework.
2.4.1 Constraints for the Optical Flow from the Intensity Conservation
One way of deriving the gradient constraint is based on image intensity conservation, stating that the image intensity is constant along the motion trajectories (see [38] for details):
Applying the chain rule:
t
Since we have two unknowns and one constraint equation, it is common to combine constraints from several neighboring locations:
( )
2I, with corresponding lower indices, stands for spatial (Ix, Iy) and temporal (It) derivatives of the image. This equation assumes that motion vectors are distributed smoothly (it is naturally not always true). W is a decaying window (e.g. Gaussian form) and E(Vx,Vy) can be considered as a measure of how much the set of constraints are satisfied by a motion vector candidate.
Giving an analytical expression for the velocity vectors in the least square estimate sense we can write:
0
Leaving small indexes the above equations can be written in a matrix form:
0 Assuming that M is invertible the solution is given:
b M
Vˆ =− −1 (13)
This description uses a constant model for V in a small neighborhood, giving further constraints to solve the problem in the least square sense. As to find the most appropriate motion vectors, we must obviously choose the vector that approximates the constraint equation with the least error.
It is easy to see that the elements of M and b can be directly computed in a parallel way, well suited to our cell array framework and this calculation is required only once for each video frame. The evaluation of a motion vector candidate in each pixel position at time t, carried out by some multiplications and additions with the elements of M and b, foretells that the evaluation can be a part of a motion optimization/segmentation algorithm. This way, motion vector estimation and segmentation can be combined into one model and can be processed by the iterative change of Vx and Vy and evaluating an energy function defined below.
2.4.2 Segmentation by Energy Minimization
To achieve our final goal of spatio-temporal video segmentation it is not necessary to achieve a very accurate optical flow, but rather to get a smooth segmentation of the moving areas. Next we propose a segmentation model that is based on the energy minimization of a formulae including the smoothness of the optical flow and the gradient-based motion estimation model.
The optical flow segmentation model is similar to the previous MRF based image segmentation. Now, the energy term of this model to be minimized is the following:
∑ ⋅ + + ∑
In the equations ω is the appropriate label field and Vωsis the corresponding velocity candidate. As we want to optimize a vector field, there might be too much possible vector candidates (in other words in general motion fields the number of possible classes could be too large) so it would make it impossible to achieve fast convergence.
Due to this reason we made two restrictions on the label field (similarly to the segmentation method in the previous section):
1. For the initial vector field we use the vectors obtained by an initial estimation.
During the optimization no new values are introduced.
2. In the optimization process a label can be changed only to the value of one of its neighbors.
We found other authors applying the same constraints with success under similar conditions [11,40].
With these restrictions fast convergence can be reached within some hundred iterations using the MMD optimization method. Similarly to the previous motion segmentation model, this algorithm also requires 2 cell array layers for the state representation and optimization of Vx and Vy respectively.
Fig. 7 (a) and (b) shows the results of segmentation on the input images Fig. 5 (b) and (c) respectively. Fig. 8 shows motion segmentation results from the sequence
“Hamburg Taxi”.
(a) (b)
Fig. 7.
Segmented vector fields with the gradient-based optimization method of input images Fig. 5 (b) and (c).
(a) (b)
Fig. 8.
(a) Segmented vector field with the gradient-based optimization method, (b) result projected onto frame #2 of the sequence “Hamburg Taxi”.
2.4.3 Related Motion Segmentation Models Based on Energy Minimization
Here we describe shortly other motion segmentation models, that we found similar to the proposed method. The most similar segmentation algorithms found in the literature are based on the model of Horn and Schunk [38], where the estimated velocity field is obtained by the minimization of the form:
( ) ( )
( )
∑ IxVx +IyVy +It 2 +λ ∇Vx 2 + ∇Vy 2 (16) where the magnitude of λ determines the influence of the smoothness term. Iterative algorithms are used to minimize Eq. (16), e.g.:
( )
however, our model simplifies computations in several aspects. In our solution the first term contains smoothing effects itself, and there is no need to reevaluate Vˆ andxkk
Vˆ during the iterations, M and b are invariant during our algorithm. While they
second terms are very similar, our smoothness constraint is justified to the Gibbs/Markov model, similarly to the optimization procedure, where our Markovian model contains a pseudo-stochastic optimization - instead of the iterative calculation of Eq. (17) - to solve the minimization problem.
Other Markovian segmentation models are described in [11,29] but with different estimation models, both referenced papers deal with global affine motion models not well-suited to our pixel-based 2D array model.