2.1.

Optical Flow Field

Research on modelling a moving crowd by optical flow is the most developed. For example, Hu et al.¹⁵ used a Gaussian adaptive resonance theory network to extract the prominent flow vectors from a dense optical flow field and then constructed a directed neighborhood graph based on the shortest path search for these flow vectors. Hierarchical agglomerative clustering algorithm was used to segment the directed neighborhood graph to obtain the final segmentation results of a moving crowd. Zhang et al.¹⁶ obtained trajectory chains of feature points using an orientation distribution function in a sparse optical flow field; then, they used a spectral clustering method (number of clusters determined by prior knowledge) to complete the classification of trajectories to perform slow-motion segmentation in densely crowded locations. This type of approach usually establishes a crowd motion model directly based on the similarity among optical flow vectors. Because the model is relatively simple, it is generally only applicable to a crowd with good motion consistency.

2.2.

Particle Flow Field

Particle flow field is based on the Lagrangian fluid dynamics framework. When used for motion segmentation, the particle trajectory can be estimated by numerical integration through the moving particle grid in the optical flow field. For example, Ali and Shah²² first used the Lagrangian particle dynamic (LPD) algorithm to estimate the particle flow field based on the optical flow field, and then used the finite time Lyapunov exponent to extract the boundary among different flow fields to obtain the segmentation results for a crowd. Because a particle flow field ignores the spatial domain variation and the time delay is obvious, a crowd motion segmentation method based on particle flow field is only suitable for a high-density slow-moving crowd.

2.3.

Streak Flow Field

To obtain a better flow-based model for a moving crowd, Mehran et al.²⁴ introduced streak line in fluid dynamics to estimate the motion field of a crowd, which is called streak flow. Because streak flow can preserve the motion information of the flow for a period, the segmentation performance is better than particle flow when applied to crowds with obvious motion changes. However, the streak flow field is obtained on the basis of the optical flow field. The calculation process of streak flow requires very high accuracy optical flow; however, it is very difficult to meet accuracy requirements for optical flow using conventional methods.

2.4.

Vector Domain Segmentation for Flow Field Model Application

To make the flow field model suitable for crowds with different densities and motion modes, some researchers have used the vector domain segmentation model^36,37 in the vector field to establish a flow-based model of a moving crowd, such as the LTDS model proposed by Wu and Wong²⁰ Through the normalization of the vector magnitude and the addition of foreground area constraints, LTDS extends the TDS model proposed by Roy et al.³⁶ to the nonunit field; it can be applied to the segmentation of crowds with different densities by localization. However, because the objective function adopted by LTDS only considers the direction difference between vectors, undersegmentation between foreground and background can easily occur.

The proposed crowd motion segmentation method is based on a segmentation model in the optical flow field. The ICS measure is used to replace the similarity measure based on normalized vector inner product in the objective function of LTDS. The improved objective function can distinguish between both magnitude and direction of vectors to achieve better segmentation performance.

3.1.

TDS and LTDS

Given a unit vector field $E$, the translation domain (TD) is defined as the region in which all vectors have the same direction. The field lines in a TD are a set of parallel lines. Therefore, there must exist a unique normal vector perpendicular to the field lines, denoted as $a$, which can be used as a characteristic parameter to represent the TD. $a$ is also known as the dominant translation parameter. Roy et al.³⁶ proved that a vector domain is a TD if and only if there is a unique vector $a(\mathrm{\Omega })$, for $\forall x\in \mathrm{\Omega }$, that satisfies the following equation:

On the basis of Eq. (2), given a deformation velocity $V$, the Gâteaux derivative in the direction of $V$ can be given as follows:

Meanwhile, Roy et al.³⁶ also proposed an estimation method for the dominant translation parameter $a(\tau )$, which takes the first half of the right side of Eq. (2) as the objective function and obtains the optimal estimate of $a(\tau )$ by minimizing the objective function, as follows:

Because the TDS model is proposed for the TD in the ideal unit vector field, it cannot be directly applied to the segmentation of an actual motion field. To make the TDS model applicable to crowd motion segmentation, Wu and Wong²⁰ established a LTDS model by the normalization of the vector, addition of a foreground motion region area constraint, and localization according to Eq. (2). The objective function of LTDS is as follows:

Similar to the evolution equation [Eq. (4)] of TDS, and considering that $\mathcal{F}[A(x),x]$ is an even function, which makes it impossible to distinguish two completely opposite vectors, the evolution equation of the active contour algorithm corresponding to the LTDS model is as follows:

3.2.

ICS

It is well known that the cosine similarity (CS) measure uses the cosine of the angle between two vectors in the vector space to measure the difference between these two vectors:

To measure both magnitude and direction difference between the two vectors using CS, we first add a scale factor that can measure the difference between the magnitudes of the two vectors in the original CS, as shown in Eq. (8); this is defined as follows:

Considering that the range of $k_M(\mathit{x},\mathit{y})$ is [0, 1] and that of CS is [$-\mathrm{1,1}$], to unify the ranges of both values, by linear mapping, the CS is adjusted to

The basic ICS defined in Eq. (11) cannot be directly used for TD segmentation because there is an imbalance when evaluating the magnitude and direction difference. For example, if the similarity is given as 0.5, when $k_M\to 1$ (i.e., there is almost no difference in magnitude between the two vectors) and $\cos {\theta }_{xy}\to 0$ can be obtained from Eqs. (10) and (11), then ${\theta }_{xy}\to \pi /2$; however, when ${\theta }_{xy}\to 0$ (i.e., there is almost no difference in direction between the two vectors), ${\mathrm{CS}}_1(x,y)\to 1$, then $k_M\to 0.5$. This shows that the ICS between the two vectors with small difference in direction and half the difference in magnitude is the same as the ICS between the two vectors that are perpendicular to each other and have the same magnitude, as shown in Fig. 2(a). Considering that there are two vectors perpendicular to a given vector in the same plane, $\mathrm{ICS}(x,y_1)=\mathrm{ICS}(x,y_2)=\mathrm{ICS}(x,y_3)$.

Fig. 2

Imbalance of vectors with the same ICS defined in Eq. (11). (a) ICS between the two vectors with no difference in direction but half the difference in magnitude [i.e., $\mathrm{ICS}(x,y_1)$] is the same as the ICS between the two vectors perpendicular to each other with same magnitude [i.e., $\mathrm{ICS}(x,y_2)$ and $\mathrm{ICS}(x,y_3)$]. (b) An optical flow field containing the optical flow vectors having the relationship described in panel (a) (the vector in red is the given reference vector $x$).

For the actual application of this study, $\mathrm{ICS}(x,y_1)$, $\mathrm{ICS}(x,y_2)$, and $\mathrm{ICS}(x,y_3)$ should not be the same. Although the magnitude of ${\mathit{y}}_1$ is quite different from that of $\mathit{x}$, they can be considered to have good similarity because they have the same directional angle (i.e., they have exactly the same direction of motion). In contrast, both ${\mathit{y}}_2$ and ${\mathit{y}}_3$ are perpendicular to $x$, resulting in a large difference in their direction of motion. In this case, even if they have the same magnitude, the similarity between them should be very low. Otherwise, two vectors with vertical or even opposite directions may be segmented into the same region, i.e., all the vectors in Fig. 2(b) will be segmented into the same region. That is, when segmenting vectors, the importance of the similarity between the directional angles is significantly higher than the similarity between the magnitudes; for the case shown in Fig. 2, the condition $\mathrm{ICS}(x,y_1)\gg \mathrm{ICS}(x,y_2)=\mathrm{ICS}(x,y_3)$ should be satisfied. To enhance the proportion of directional angle similarity in the basic ICS defined in Eq. (11), we add an exponential adjustment factor $\beta$ to ${\mathrm{CS}}_1(x,y)$ that measures the directional angle difference in ICS; then ICS is redefined as follows:

Fig. 3

Relationship between ${\mathcal{R}}_{\theta }^{\beta }$ and $\beta$. When the minimum similarity ${\mathrm{s}}_{\min }$ is given, i.e., $\mathrm{ICS}(x,y,\beta )\ge s_{\min }$, the larger $\beta$ is, the smaller will be ${\mathcal{R}}_{\theta }^{\beta }$.

The value of $\beta$ can be determined according to the distribution of the vectors in the optical flow field, especially the distribution of the vector direction by using the maximum allowable ${\mathcal{R}}_{\theta }^{\beta }$ at minimum similarity ${\mathrm{s}}_{\min }$. For example, the minimum similarity ${\mathrm{s}}_{\min }$ is given as 0.5 and ${\mathcal{R}}_{\theta }^{\beta }$ is given as $\pi /6$ at this similarity [as shown in Fig. 4(a)]; when $k_M=1$, ${[{\mathrm{CS}}_1(x,y)]}^{\beta }$ takes the minimum value of 0.5 and the corresponding direction angle ${\theta }_{xy}$ has the maximum value of $\pi /6$, so the following equation holds as follows:

Fig. 4

Determination of $\beta$ and its effect given ${\mathrm{s}}_{\min }=0.5$ and ${\mathcal{R}}_{\theta }^{\beta }=\pi /6$ corresponding to ${\mathrm{s}}_{\min }$. (a) From $\mathrm{ICS}(x,y_1,\beta )=\mathrm{ICS}(x,y_2,\beta )=\mathrm{ICS}(x,y_3,\beta )$, $\beta \hspace{0.17em}=\hspace{0.17em}9.9969$ can be determined. (b) After $\beta$ is determined, the values of ${\mathrm{k}}_{\mathrm{M}}$ and ${\theta }_{xy}$ are both limited to the shaded area.

3.3.

ICS-Based TDS and ICS-Based LTDS

It can be seen from Eqs. (2) and (6) that $\mathcal{F}[\mathit{A}(\mathit{x}),\mathit{x}]$ in the objective function of LTDS is similar to the term ${\int }_{\mathrm{\Omega }(\tau )}{[\mathit{a}(\tau )·\mathit{E}(\mathit{x})]}^2\mathrm{d}\mathit{x}$ in the objective function of TDS; they essentially use the CS to measure whether the vector to be segmented is perpendicular to the dominant translation parameter $\mathit{a}(\mathrm{\Omega })$ of $\mathrm{\Omega }$. In the actual application process, when $\mathrm{\Omega }$ is constantly evolving, the degree of its unit TD approximation becomes worse; the difference between the magnitudes of the vectors may also increase. In this case, if the magnitude difference is not equalized due to improper selection of $\mu$, undersegmentation may occur, i.e., regions with vectors having small direction difference but large magnitude difference may be segmented into one region, as shown in Fig. 5. In an actual motion field, the difference between magnitude is often an important basis for distinguishing the foreground from the background; however, the error constraints in the objective function of both TDS and LTDS do not consider the magnitude difference between vectors, which causes the segmentation results to include more background in the motion regions.

Fig. 5

Undersegmentation caused by the magnitude difference between foreground and background. In the left, the vectors of foreground (gray shaded region) and background (green shaded region) have no difference in terms of direction, but they have obvious difference in magnitude. When the initial region $\mathrm{\Omega }$ of the active contour algorithm is inside ${\mathrm{\Omega }}_1$ (as indicated by the red dashed line on the left), if the magnitude difference is not equalized due to improper selection of $\mu$, the evolution result ${\mathrm{\Omega }}^{\prime }$ (the area within the red border on the right) will contain both the foreground and the background, i.e., ${\mathrm{\Omega }}^{\prime }={\mathrm{\Omega }}_1\cup {\mathrm{\Omega }}_2$.

Because ICS can measure the difference between both direction and magnitude of vectors, it is expected to improve the segmentation performance by improving the objective function of LTDS and TDS. The key is to use ICS to redefine the error constraint term in the objective function of LTDS and TDS.

Referring to Eq. (2), the objective function of the TDS redefined by ICS (ICS-TDS) is as follows:

After defining the objective function of ICS-TDS given by Eq. (14), referring to the localization method in Eq. (6), we can obtain the objective function of ICS-based LTDS (ICS-LTDS) by redefining $\mathcal{F}(\mathit{A}(\mathit{x}),\mathit{x})$ with ICS:

Similar to TDS and LTDS, the minimization of the objective functions in Eqs. (14) and (16) can also be performed by the active contour algorithm. The evolution equation of ICS-TDS can be obtained by rewriting Eq. (4) as follows:

As ${\mathcal{F}}_{\text{ICS}}[\mathit{B}(\tau ,\mathit{x}),\mathit{x}]$ is not an even function, opposite direction vectors can be well distinguished. Therefore, the evolution equation of the active contour algorithm corresponding to ICS-LTDS can be rewritten by referring to Eq. (7):

3.4.

Acquisition of Initial Regions

As we know, for the active contour algorithm, the selection of the initial regions has a significant impact on the final segmentation results. The segmentation results obtained by erroneous initial regions may be intertwined with the correct segmentation results; therefore, they cannot be eliminated. To obtain suitable initial regions of evolution, Wu and Wong²⁰ first selected the minimum point $p$ in the LMCM and then calculated the mean ${\overline{\mathcal{F}}}_p$ of $\mathcal{F}$ and the mean optical flow magnitude ${\overline{M}}_p$ in the neighborhood of $p$. Only the neighborhood of a point with ${\overline{\mathcal{F}}}_p$ and ${\overline{M}}_p$ higher than given thresholds, which indicates that the motion consistency in its neighborhood is good enough and the possibility that its neighborhood is the foreground is sufficiently high, can be considered as the initial region. However, when this initial region is used for medium- or low-density crowds, if the optical flow estimation error leads to some larger magnitudes of optical flow in the background, some erroneous initial regions may be selected in the background, as shown in Fig. 6(b). In Fig. 6(b), the initial regions in the red circle are all erroneous initial regions.

Fig. 6

Example of the acquisition of initial regions. (a) The minimum (red points) extraction results in LMCM; (b) The initial regions extracted by ${\overline{\mathcal{F}}}_p$ and ${\overline{M}}_p$, where the initial regions within the red circle are the erroneous initial regions in the background; and (c) The initial regions extracted after the introduction of ${\overline{D}}_p$, where all the erroneous initial regions in (b) are removed.

Although the magnitudes of optical flow in these erroneous regions are relatively large, the gray level changes in these background regions are actually small and can be easily estimated by the frame difference method. Therefore, in addition to ${\overline{\mathcal{F}}}_p$ and ${\overline{M}}_p$, we introduce the regional mean ${\overline{D}}_p$ based on the frame difference for initial region determination. Only when ${\overline{\mathcal{F}}}_p$, ${\overline{M}}_p$, and ${\overline{D}}_p$ are all higher than given thresholds, which are, respectively, called local error threshold ${\mathrm{TH}}_{\overline{\mathcal{F}}}$, local motion magnitude threshold ${\mathrm{TH}}_{\overline{M}}$, and local frame difference magnitude threshold ${\mathrm{TH}}_{\overline{D}}$, the neighborhood of the minimum point $p$ can be confirmed as the initial region. The initial regions extracted after the introduction of ${\overline{D}}_p$ can be seen in Fig. 6(c). By comparing Figs. 6(b) and 6(c), it can be seen that most of the erroneous initial regions in the background are removed.

3.5.

Algorithm Flow of the Proposed Method

As mentioned above, ICS-based LMCM is more suitable for segmenting high-density crowds with slow consistent motion because it is based on the local ICS error constraints that can well reflect the motion consistency between local regions. In contrast, ICS-based GMCM is based on the ICS error constraints between all vectors in the motion field and a given TD; therefore, it is more suitable for segmenting motion regions that are consistent with the given TD. If the initial regions contain all possible motion modes, the GMCMs corresponding to all the initial regions can be used to segment the moving crowd with medium- or low-density and complex motion modes.

In summary, the detailed flow of the proposed crowd motion segmentation algorithm based on ICS-LTDS/ICS-TDS is shown in Algorithm 1.

Algorithm 1

Proposed algorithm.

4.1.

Segmentation of High-Density Slow-Moving Crowds

Algorithm 1 is applied for segmentation of high-density slow-moving crowds, as described in Sec. 3.5, in the LMCM by using the evolution equation [Eq. (19)] based on the objective function of ICS-LTDS ($\beta =9.9969$). In all experiments, if the size of the entire image is mapped to $[\mathrm{0,1}]\times [\mathrm{0,1}]$, the size of the neighborhood $\ell (x)$ is $0.02\times 0.02$. The initial region of evolution is centered on the minimum of the LMCM and the size is $5\text{pixels}\times 5\text{pixels}$. The thresholds ${\mathrm{TH}}_{\overline{\mathcal{F}}}$ and ${\mathrm{TH}}_{\overline{M}}$ are set to 0.06 and 2, respectively. When the threshold ${\mathrm{TH}}_{\overline{D}}$ is used for different image sequences, the set values are between 1.3 and 3.4. The window size for calculating ${\overline{\mathcal{F}}}_p$, ${\overline{M}}_p$, and ${\overline{D}}_p$ is $5\text{pixels}\times 5\text{pixels}$. The constant $\gamma$ in $\mathcal{G}$ is set to 0.1. The constant $\mu$ is given by referring to LTDS:²⁰

Fig. 7

Image sequences of high-density slow-moving crowds used for testing: (a) Mecca, (b) pilgrims, (c) marathon, and (d) kabba. Representative segmentation results (top to bottom): Ground truth images, segmentation results of the proposed ICS-LTDS, segmentation results of LTDS, segmentation results of LPD, segmentation results of streak flow, and segmentation results of prior knowledge-based trajectory tracking.

Table 1

Segmentation accuracies of the proposed ICS-LTDS, LTDS, LPD, streak flow, and prior knowledge-based trajectory tracking for the scenes in Fig. 7.

To compare the segmentation results of the proposed ICS-LTDS algorithm with those of the typical nonflow-based methods in scenarios with high-density slow-moving crowds, we selected pilgrims and marathon as test sequences, and applied the proposed ICS-LTDS, TCA,³⁴ and the CF-based algorithm³² to them. Representative segmentation results obtained from them are shown in Fig. 8 and the segmentation accuracies calculated according to Eq. (20) are shown in Table 2. For TCA, the following parameter settings were used: $p=16$, $r=40$ (10 for pilgrims), $\tau =30$, $\mathrm{\Delta }=100$, $\delta =1$, $\alpha =0.1$, $\beta =20$, and $\gamma =150$; for the CF algorithm, $K=20$, $z=0.025$, the upper bound of $\mathrm{\varphi }$ was taken as 1, and the threshold coefficient $\alpha$ was 0.5.

Fig. 8

Image sequences of high-density slow-moving crowds used for testing (top to bottom): pilgrims and marathon. Representative segmentation results: (a) ground truth images; (b) segmentation results of the proposed ICS-LTDS; (c) segmentation results of TCA; and (d) segmentation results of CF algorithm.

Table 2

Segmentation accuracies of the proposed ICS-LTDS, TCA, and CF algorithm for the scenes in Fig. 8.

Similar to crowd flow in high-density slow-moving crowds, the proposed ICS-LTDS algorithm can also be applied to slow-moving traffic flow or mixed flow. We selected two image sequences for algorithm evaluation in this noncrowd scenario. In addition, flow-based ICS-LTDS, LTDS, and LPD, and nonflow-based TCA were selected for the evaluation and comparison in the same noncrowd scenario. Representative segmentation results for two image sequences of noncrowd scenes are shown in Fig. 9. The segmentation accuracies calculated according to Eq. (20) are shown in Table 3.

Fig. 9

Image sequences of noncrowd scenes used for testing (top to bottom): roundabout and highway traffic. Representative segmentation results: (a) ground truth images; (b) segmentation results of the proposed ICS-LTDS; (c) segmentation results of LTDS; (d) segmentation results of LPD; and (e) segmentation results of TCA.

Table 3

SA of the proposed ICS-LTDS, LTDS, LPD, and TCA for the scenes in Fig. 8.

Further, we obtained the ground truth of 100 consecutive frames from the marathon sequence manually to perform evaluation on a sequence containing a slow-moving crowd. The segmentation results of the representative five consecutive frames of the three best-performing flow-based algorithms (i.e., the proposed ICS-LTDS, LTDS,²⁰ and LPD²²) and the nonflow-based CF algorithm³² are shown in Fig. 10. The highest, lowest, and average segmentation accuracies of all 100 frames in the entire sequence are shown in Table 4.

Fig. 10

Segmentation results of the representative five consecutive frames (top to bottom) in the marathon sequence: (a) ground truth; (b) segmentation results of the proposed ICS-LTDS; (c) segmentation results of LTDS; (d) segmentation results of LPD; and (e) segmentation results of CF algorithm.

Table 4

Highest, lowest, and average segmentation accuracies of all 100 frames in the Marathon sequence.

From the results in Figs. 7–10 and Tables 1–4, the following are clear: (1) As the principles of particle flow and streak flow are similar, both process motion information recorded for a period; therefore, the segmentation results of LPD and streak flow are similar. The proposed ICS-LTDS is an improvement of LTDS and so the segmentation results of these two methods are similar. (2) Although particle flow and streak flow are highly suitable for segmenting slow-moving targets, when the speed of the target has a downward trend and tends to stop, both methods will assume that the target has not moved for a period and thus abandon the extraction of the target. For example, in the highway traffic sequence (last row in Fig. 9), the traffic that decelerates and gradually stops because of congestion is not segmented by LPD. In contrast, as ICS-LTDS and LTDS are based on real-time optical flow, both segment these targets. (3) As the prior knowledge-based trajectory tracking method is used in the sparse optical flow field formed by feature points, it is only suitable for the motion segmentation of people in extremely crowded places. For a scene where the feature points are not easily extracted or the crowd has looser distribution, missed segmentation is highly possible, as shown in the last row of Figs. 7(a)–7(d). (4) From the discussion in Sec. 3.3, as the objective function of LTDS only distinguishes the direction differences between vectors regardless of the magnitude differences, it is prone to produce undersegmentation results between foreground and background. This phenomenon is particularly evident in the segmentation results for the roundabout and highway traffic sequences in Fig. 9(c), and the marathon sequence in Fig. 10(c), which results in a lower SA than ICS-LTDS with improved objective function. (5) As TCA needs to record the complete trajectories of the keypoints, it cannot cope with short-term changes of the moving regions. This makes it easy to produce undersegmentation between the foreground and background, as shown in Figs. 8(c) and 9(e). (6) Except for the pilgrims and kabba sequence, the SA of ICS-LTDS is higher than other methods, indicating that the ICS-LTDS algorithm is more suitable for segmentation of high-density slow-moving crowds than existing flow-based and nonflow-based crowd segmentation algorithms.

4.2.

Segmentation of Medium- or Low-Density Crowds with Complex Motion Modes

Algorithm 1 is applied to medium- or low-density crowds with complex motion modes, as described in Sec. 3.5, in the GMCM corresponding to each initial region by using the evolution equation [Eq. (18)] based on the objective function of ICS-TDS ($\beta =9.9969$). In all experiments, the average vector of the initial region is taken as $\widehat{b}(\tau )$ to calculate the GMCM. Other parameters, such as ${\mathrm{TH}}_{\overline{\mathcal{F}}}$, ${\mathrm{TH}}_{\overline{\mathrm{M}}}$, ${\mathrm{TH}}_{\overline{D}}$, $\gamma$ and $\mu$, are all consistent with ICS-LTDS.

In LTDS, the evolution of the objective function is performed in LMCM, which is quite different from ICS-TDS. Therefore, in addition to LTDS, segmentation results of TDS, which also performs the evolution in the GMCM, are included for comparison. However, as the basic TDS proposed in Ref. 36 is only applicable to the unit vector field, it cannot be directly used in the actual scenes, as shown in Fig. 10. To make TDS available for crowd segmentation in the actual motion field, by referring to LTDS, we improve TDS by normalizing the motion vector and adding the foreground motion area constraint in the objective function of TDS:

In addition, among other methods based on flow field modeling, the particle flow-based LPD is selected for algorithm comparison. To facilitate the comparison among algorithms, the common parameters of ICS-TDS, TDS-$\mathcal{G}$, and LTDS are given the same values. The representative segmentation results of ICS-TDS, TDS-$\mathcal{G}$, LTDS, and LPD applied to four image sequences are shown in Fig. 11, and the segmentation accuracies are shown in Table 5.

Fig. 11

Representative segmentation result. Image sequences used for testing (top to bottom): crosswalk, UCSD pedestrians, Fudan pedestrians, and multidirectional pedestrians (MD pedestrians). (a) Ground truth images; (b) segmentation results of proposed ICS-TDS; (c) segmentation results of TDS-$\mathcal{G}$; (d) segmentation results of LTDS; and (e) segmentation results of LPD.

Table 5

SA of the proposed ICS-TDS, TDS-G, LTDS, and LPD for the scenes in Fig. 11.

It should be noted that in a close-up scene containing multidirectional pedestrians as shown in the fourth row of Fig. 10, the term $T_i$ in Eq. (20) is difficult to estimate due to the numerous moving directions of the pedestrians, mutual occlusion, and even differences in moving directions of the different parts of the same body, which makes it impossible to calculate the SA separately for different directions according to Eq. (20). Considering that the SA should simultaneously measure the similarity between both area and direction of the segment and the ground truths, on the basis of Eq. (20), we add the direction consistency measure between ${\mathrm{\Omega }}_i$ and ${\mathrm{\Omega }}_i\cap T$, so that the mean of the SA for different directions can be directly calculated without segmenting the ground truth $T$. The SA after adding the direction consistency measure is as follows:

To compare the segmentation results of the proposed ICS-LTDS algorithm with those of the typical nonflow-based methods in crowds with medium- or low-density and complex motion modes, we selected the crosswalk and MD pedestrians as test sequences, and applied the proposed ICS-LTDS, TCA,³⁴ and the CF-based algorithm³² to them. Representative segmentation results are shown in Fig. 12 and the segmentation accuracies calculated according to Eq. (20) [Eq. (26) for MD pedestrians] are shown in Table 6. For TCA, the parameter settings were as follows: $p=16$, $r=40$, $\tau =30$, $\mathrm{\Delta }=100$, $\delta =1$, $\alpha =0.1$, $\beta =20$ and $\gamma =150$; for the CF algorithm, $K=20$, $z=0.025$, the upper bound of $\mathrm{\varphi }$ was taken as 1, and the threshold coefficient $\alpha$ as 0.5.

Fig. 12

Representative segmentation result. Image sequences used for testing (top to bottom): crosswalk and MD pedestrians. (a) Ground truth images; (b) segmentation results of proposed ICS-TDS; (c) segmentation results of TCA; and (d) segmentation results of CF algorithm.

Table 6

Segmentation accuracies of the proposed ICS-TDS, TCA, and CF algorithm for the scenes in Fig. 12.

Similar to Sec. 4.1, we obtained the ground truth of the 100 consecutive frames in the crosswalk sequence manually to evaluate sequences containing fast-moving crowds with complex motion modes. The segmentation results of the representative five consecutive frames of the three best-performing flow-based algorithms (i.e., the proposed ICS-TDS, TDS-$\mathcal{G}$,³⁶ and LPD²²) and the nonflow-based CF algorithm³² are shown in Fig. 13. The highest, lowest, and average segmentation accuracies of all 100 frames in the entire sequence are shown in Table 7.

Fig. 13

Segmentation results of the representative five consecutive frames (top to bottom) in the crosswalk sequence: (a) ground truth; (b) segmentation results of proposed ICS-TDS; (c) segmentation results of TDS-$\mathcal{G}$; (d) segmentation results of LPD; and (e) segmentation results of CF algorithm.

Table 7

Highest, lowest, and average segmentation accuracies of all 100 frames in the crosswalk sequence.

From the results shown in Figs. 11–13 and Tables 5–7, the following is clear: (1) The particle flow-based LPD algorithm is not suitable for segmenting crowds with loose distribution, especially for segmenting crowds with complex motion modes. For example, in the segmentation results of crosswalk, as shown in the first row in Fig. 11(e) and the last row in Fig. 13(d), there are obvious missing or incorrect segmentations. Meanwhile, in the segmentation results of multidirectional pedestrians, several pedestrians with distinctly different motion directions are segmented in the same region, resulting in severe undersegmentation [as shown in Fig. 14, which is the zoomed-in version of the last row of Fig. 11(e)]. (2) As LMCM is based on the motion consistency between local regions, when pedestrians with little difference in motion direction occlude each other, their motion boundaries are not clear enough in LMCM, which makes LTDS prone to produce undersegmentation results for pedestrians that occlude each other, as shown in Fig. 15 [the zoomed-in version of the last row of Fig. 11(d)]. (3) Although TDS-$\mathcal{G}$ and LTDS have almost the same objective function, TDS-$\mathcal{G}$ evolves in the GMCM while LTDS evolves in the LMCM. Except for UCSD pedestrians, the segmentation results of TDS-$\mathcal{G}$ are better than those of LTDS. It is indicated that the method based on the evolution in the GMCM is more suitable for segmenting medium- or low-density crowds with complex motion modes. As TDS-$\mathcal{G}$ has the same objective function as LTDS, which only distinguishes the vector direction but not the vector magnitude, the undersegmentation results between foreground and background caused by the objective function are still serious in the scene shown in Figs. 11 and 13; therefore, their SA is significantly lower than that of ICS-TDS. (4) As TCA cannot cope with short-term changes in the moving regions, when it is applied to crowd scenes containing complex motion patterns, the segmentation results still contain severe undersegmentation, as shown in Fig. 16(b) [zoomed-in version of the last row of Fig. 12(c)]. Meanwhile, as the coherent neighbor invariance mainly focuses on the pairwise motion consistency and ignores the motion difference among all the keypoints in the local region of the center point, when the CF algorithm is applied to crowd scenes containing complex motion patterns, undersegmentation results are easily produced, as shown in Fig. 16(c) [zoomed-in version of the last row of Fig. 12(d)]. (5) For the four scenes shown in Fig. 10, ICS-TDS is superior to other methods in terms of both segmentation effect and SA, indicating that the ICS-TDS algorithm is more suitable for segmenting medium- or low-density crowds with complex motion modes than existing flow-based and nonflow-based crowd motion segmentation algorithms.

Fig. 14

Undersegmentation results produced by LPD: (a) ground truth and (b) segmentation results of LPD. In panels (a) and (b), the same ellipse corresponds to the same pedestrians. In the ground truth, the pedestrians in blue and yellow ellipses have at least two motion directions, but in the segmentation results of LPD, the pedestrians in each ellipse have the same motion direction.

Fig. 15

Undersegmentation results produced by LTDS. (a) Ground truth and (b) segmentation results of LTDS. In panels (a) and (b), the same ellipse corresponds to the same pedestrian(s). In the ground truth, the pedestrians in blue, yellow, and red ellipses have two motion directions, but in the segmentation results of LTDS, the pedestrians in each ellipse have the same motion direction.

Fig. 16

Undersegmentation results produced by TCA and CF algorithm. (a) Ground truth; (b) segmentation results of TCA; and (c) segmentation results of CF algorithm. In panels (a), (b), and (c), the same ellipse corresponds to the same pedestrian(s). In the ground truth, the pedestrians in blue, yellow, and purple ellipses have at least two motion directions, but in the segmentation results of TCA and CF algorithm, the pedestrians in each ellipse have the same motion direction.