Evaluating Crowd Density Estimation Models

This post is an extension of the crowd density estimation  post. The models used here were trained using almost no hyperparameter tuning so, please ignore their actual performance. It is possible to improve the performance by carefully selecting the image augmentation(s), learning rate, learning rate schedule, and the optimizer. This post's objective is to visualise the predictions across different variables to understand the model performance, and identify areas that can be improved.

The vega plots displayed on this page are pulled from the GitHub and might take some time to load depending on your network connection.

SECTION 1: METRICS, PREDICTIONS & INPUT SIZES

In this section, we are going to plot the predictions of models based on two different architectures: VGG16 Baseline and VGG16+Decoder model. These predictions are generated by 4 different models:

• ➤ VGG16 Baseline: trained on 224px images
• ➤ VGG16 Baseline: trained on 448px images
• ➤ VGG16+Decoder: trained on 224px images
• ➤ VGG16+Decoder: trained on 448px images

Each plot displays two overall metrics: Mean Squared Error and Mean Absolute Error. We are using two different plots in this section.

1: Scatter Plot

2: Scatter + Heatmap

We first start with the baseline architecture and then present the decoder based architecture. Each sub-section presents the question, and plots provide the potential answer, if any.

VGG16 BASELINE

VGG16 + DECODER

We repeat the same for VGG16 + Decoder based architecture. Note that we are still not making any direct comparison between two different architectures used here.

SECTION 2: BASELINE vs DECODER

We now move to a direct comparison between two different architectures. We found above that increasing the input size of training data improves the model performance. We use the predictions from the models trained on 448px images from now on.

We continue plotting predictions on images of three different input sizes, but this time we plot train and test set together. The plot on the left shows the baseline prediction, and the right one shows that of decoder. Here, you can highlight a section of the plot and make a direct comparison between the baseline and decoder model.

SECTION 3: CROSS-FILTERING & IMAGE CLUSTERS

And in the final section, we use the power of cross-filtering to review specific predictions across different models as well as the image clusters. We use the dimensionality reduction algorithm, t-SNE in this case, to cluster the images.

The plots below are generated using the baseline and decoder based models trained on 448px images. We show model predictions on 224, 448, and 600 px images in the above sections. However, here I am just using the predictions on 448px images, mainly because we want to highlight the approach. We could very well do this for different input sizes as well as the datasets if required.

You can select a part of the plot, and see other two plots highlight the relevant points/images. Go ahead, give it a try.

Train set: Use the mouse to select the area

The model was trained on 448px images, and we see that both the models can overfit the training data, which is something that you would want to do before you start regualising the model. Now, let's review the test data.

Test set: Use the mouse to select the area

Clearly, we have some work to do to get the model to generalise better. We also need to review the4 green dotsthat have the crowd density between 600-1000 (see x-axis). These images do not come from the same cluster.

And, although there aren't nice and clear clusters in this particular dataset. The idea seems quite useful to me. It can help me answer questions such as do certain clusters perform worse than the others; if so, I can then review images in that cluster manually.