5.3 Pixel-level Motion Direction Models
Our pixel-level detectors (denoted as low-level detectors on Fig. 5.2) are to classify motion as certain or uncertain and usual or unusual, where by uncertain we mean that the motion information is unreliable. If we detect motion at a place where no motion was observable previously the algorithm should classify this case to the uncertain (but not to the unusual) group.
Also in our approach the motion of dierent image areas is considered independently from each other, so our detection method does not investigate the co-occurrence of events, which would require a global clustering of motion information. The problem with several approaches is that they assume a certain rhythm of the trac which is true for only a limited number of situations, which will be discussed in Sec. 5.4.
The performance, behavior, and output of a system based on noisy and unreliable obser-vations of thousands of data greatly depends on data collection, cleaning/ltering, interpretation and the motion model itself. In the following subsections we describe our pixel-based motion di-rection modeling techniques, starting from the interpretation of raw motion statistics, and moving on to the inclusion of temporal and spatial support. The algorithms of the following subsections investigate local motion information in a small window of time.
5.3.1 Empirical Probability Estimates of Motion Direction
One may think that the local statistics of motion directions can give valuable information about the typical motion in road environments. To investigate this, rst we simply collected 8-bin motion direction histograms for all image pixels, where the bins represented predened disjoint direc-tion classes{E,NE,N,NW,W,SW,S,SE}, each having a constant∆ = 45◦ range (see Fig. 5.4).
Formally the direction class dk is represented by its mean mk = k·45◦, where 0 ≤ k ≤7, and D = {dk}0≤k≤7. Thus the empirical model is parameterized by θE = {|dk|}0≤k≤7, where |dk|
N NW W
SW S
SE E NE
Figure 5.4: Motion vector directions are classied into eight predened direction classes.
5.3.1. Empirical Probability Estimates of Motion Direction 66
denotes the number of observations (frequency) collected during a training phase. Increasing the number of bins could enhance the precision but would also increase the uncertainty (since the learning time is limited and there is no guarantee to get a continuous distribution during learning).
Initially, we applied no spatial or temporal ltering for the collected data, but supposed that the relative occurrence of motion vectors gives a simple but eective estimate of the empirical proba-bility. Letotdenote a particular motion direction at timet, i.e.ot∈R, and dk ∈Dthe direction class whereot falls into, i.e.mk−∆/2 ≤ot≤mk+∆/2. The empirical probability ofot given the modelθE is
P(ot|θE) = |dk| P
di∈D
|di| , (5.1)
In Fig. 5.5 (b) and (c) we used eight distinct colors (Ck) for the visualization of motion directions.
The color of a pixel atx= (x, y)is calculated as a weighted sum:
RGBx= X
0≤k≤7
Px(mk|θE)Ck , (5.2)
wherePx(·|θE) denotes the empirical probability at positionx= (x, y). Naturally, there can be regions where no motion was detected in the video during the learning phase. We excluded those regions from the detection where less than 2% of the frames showed any motion during training.
(a) (b) (c) (d)
Figure 5.5: (a) sample frame from the One-Way video; (b) pixel-wise motion direction statistics;
(c) Mean Shift segmented statistics; coloring is done by Eq. 5.2; (d) an object is moving against the trac in a one-way street; red overlay is used for representing the unusually moving object detected by the Mean Shift based method, this object is also marked with a yellow rectangle.
5.3.2. Mixture of Gaussians by Expectation Maximization 67
5.3.2 Mixture of Gaussians by Expectation Maximization
In Sec. 5.3.1 a simple histogram based empirical probability model with predened direction classes was used, where the exact typical directions were unknown. Now we create a continuous model as a renement, which is able to represent multiple usual directions (i.e. multimodal). A straightforward approach is to t MoG (introduced in Sec. 2.2) on the noisy observation data. A mixture ofM Gaussians is dened as:
p(ot) =
M
X
k=1
ωkN(ot|µk, Σk) , (5.3)
where ot is the observation at time t being modeled, i.e. a motion direction at a pixel position, ωk are the weights,µk and Σk are the expected value and covariance of the Gaussians. The θEM parameters of a MoG for a particular observation set{o1, o2, . . . , oN}(observed motion directions at a given pixel position) can be computed iteratively by Expectation Maximization (EM) [19] as discussed in Sec. 2.2.1.
According to [52], if an observation is within2.5σfrom the expected value of a distribution, then we consider the observation to match the distribution. Denote the set of weights of the matching distributions withW={ωm1, ωm2, . . . , ωmk}, where1≤mi ≤M. Then we dene the probability of observationotgiven the MoG modelθEM is
P(ot|θEM) = max{W} . (5.4)
5.3.3 Adaptive Mixture of Gaussians
In the previous section a MoG was tted on the observation set (motion directions) in each pixel to have a probability model. However, the pixel-wise iterative formula of the parameter estimation made the training process extremely slow. In [51] an adaptive algorithm is proposed to update the parameters of the MoG model used for motion detection: expected value, covariance and the weight of distributions are recalculated at each frame (see Sec. 4.3 for a detailed discussion). It is a question whether the adaptive updating process, as proposed in the original article, can follow the uctuation of optical ow: while in the case of background modeling the background pixels change their values relatively fast and roughly periodically, in the current case we observe recurrence in longer periods. In our experiments we made a random initialization of the distributions (testing with dierent numbers of classes) then in the detection phase we used the method of [51] to update the MoG parameters.
Consider the same mixture ofM Gaussians as in Sec. 5.3.2. The adaptive algorithm has to
5.3.4. Spatial Support: Segmentation of Motion Statistics 68
decide if a new observationotat timetis matching with any component in the mixture. We dene the probability thatot is usual similarly to Eq. 5.4, i.e.
P(ot|θAD) = max{W} , (5.5)
and let m? = argmax{W}. At each step (at each frame) t of the training phase we update the weights for all distributions as:
ωk,t= (1−α)·ωk,t−1+α·Mk,t, (5.6)
whereαis the learning factor,Mk,tequals1ifk=m?and0otherwise. After each step the weights are normalized.
Updating of the best matching Gaussianm? is performed by modifying the expected value according to the observationot, then recalculating the covariance. This process can be formulated mathematically as
µm?,t= (1−ρ)·µm?,t−1+ρ·ot, (5.7) σm2?,t= (1−ρ)·σm2?,t−1+ρ·(ot−µm?,t)2 , (5.8) whereρ=N(ot|µm?,t−1, Σm?,t−1), or a constant value [51] as discussed in the previous chapter, and we experimentally set toρ= 0.15.
5.3.4 Spatial Support: Segmentation of Motion Statistics
To include spatial support we used the Mean Shift (MS) segmentation technique of [25] on the empirical probability estimatesP(·|θE)(see Sec. 5.3.1). The MS technique is widely used in image segmentation and has the ability to consider spatial information and other arguments such as color.
Fig. 5.5 illustrates (a) input image, (b) pixel-wise motion statistics, and (c) the segmented motion statistics map (segmentation parameters were set as proposed in [25]). We denote the set of L disjoint regions, resulted by the segmentation, byR={r1, r2, . . . , rL}. For a particular regionri, and for the pixels contained by the region, the probability distributionP(·|θMS)is obtained by the segmentation.
5.3.5. Temporal Support: Markovian Extension 69
5.3.5 Temporal Support: Markovian Extension
Although we insist on the statistical processing of raw data without object level understanding of the motion processes or without the tracking of objects, we can still use spatiotemporal information for the analysis of motion. We assume that unusual events happen not only on one frame but on at least two consecutive frames, supposing a Markov Chain property of the motion of objects. In our approach it means that if we nd an anomalously moving pixel and we estimate its motion direction at timet, then projecting back with motion compensation to the preceding frame at time t−1, here should also be a corresponding anomalous motion vector with low probability. This is formalized as follows. Let
PxU(ox,t|θ) = 1−Px(ox,t|θ) (5.9) denote the probability of a given motion directionox,tbeing unusual at timetat the pixel position x= (x, y), using the motion modelθ∈ {θE,θEM,θAD,θMS}. Then using the above assumptions:
PxUM(ox,t|θ) =PxU(ox,t|θ)× max
x0∈N(x+v){PxU0(ox0,t−1|θ)} , (5.10) where the second term of the product means that we use the highest probability value of unusual observations in theN(·)neighborhood set of the motion compensated positionx+vat timet−1, wherev= (vx, vy)denotes the optical ow velocity vector. In our experimentsN(·)was a square of size5×5around the back projected position. Our tests proved that using PUM instead ofPU makes signicant improvements in the performance of anomaly detection in case of all three motion models.
5.3.6 Results, Evaluation
To use these above methods for motion anomaly detection we have to train our statistical models.
Basically there is only one parameter to be determined: the time interval for the collection of motion directions at image pixels. We have to pay attention to gathering enough data and to avoid unusual, rare events in the training video. In our examples the number of image frames used for training was between 6000 and 8000.
We can monitor the probability of events continuously by PxU(·|θ)and PxUM(·|θ). While we basically apply pixel based processing we can still group the local estimates with a simple method:
we label all connected components (blobs above the size of 10 pixels) of the binary foreground mask with the average probabilityPUx(·|θ)or the Markovian valuePUMx (·|θ). We plot these probabilities
5.3.6. Results, Evaluation 70
(a) (b)
(c) (d)
Figure 5.6: Anomaly detection results for the frames of the One-Way sequence. (a) Empirical probability; (b) Mean Shift probability; (c) Mixture of Gaussians by EM; (d) Adaptive Mixture of Gaussians. Red dashed line: without Markovian extension (PUx(·|θ)). Blue solid line: with Markovian extension (PUMx (·|θ)).
of the most suspicious blob (the one with the highest value) as a function of frame number. Fig. 5.6 illustrates the results for the One-Way sequence where the trend lines, considered as the nal output of the detector, are the smoothed version of the probability values. In all cases the peak is at the 28500th frame, the only unusual event, when a bicycle comes in the wrong way on the one-way street. This is illustrated in Fig. 5.5(d), where the unusually moving bicycle is marked with red color. For detection we used the Mean Shift-based detector with Markovian support (PUMx (·|θMS)).
While all methods succeeded in such situations (also in other test videos), the question is the size of the gap between real unusual and other rare events. From the graphs on Fig. 5.6 it is easy to see that the Markovian extension increased the dierence between the anomalous and usual events in all cases with approximately 30%, but there is no signicant dierence between the dierent models. In general the EM method outperforms others, but unfortunately the iterative estimation of parameters (weights, expected value, and covariance) at every pixel using EM [19] is a very time consuming process and the memory requirement makes this approach unfeasible. (We have to store motion directions calculated on every frame in every pixel position. In case of 5000 training frames of a QCIF video 366MB memory is required).