What’s the Best Face Detector?. Comparing Dlib, OpenCV, MTCNN, and… | by Amos Stailey-Young | Jun, 2024

For a facial recognition disadvantage I’m engaged on, I needed to find out which facial detection model to select. Face detection is the first part of the facial recognition pipeline, and it’s very important that the detector exactly identifies faces throughout the image. Garbage in, garbage out, finally.

However, the myriad selections accessible left me feeling overwhelmed, and the scattered writings on the subject weren’t detailed enough to help me resolve on a model. Evaluating the numerous fashions took a complete lot of labor, so I figured relaying my evaluation may help folks in associated situations.

The primary trade-off when selecting a facial detection model is that between accuracy and effectivity. Nonetheless there are totally different parts to ponder.

Lots of the articles on face detection fashions are written each by the creators of the model — generally in journals — or by these implementing the model in code. In every circumstances, the writers, naturally, have a bias in the direction of the model they’re writing about. In some extreme circumstances, they’re mainly promotional advertisements for the model in question.

There aren’t many articles that consider how the fully totally different fashions perform in opposition to at least one one other. Together with further confusion, each time anyone is writing a number of model much like RetinaFace, they’re talking a number of express implementation of that model. The “model” itself is completely the neural neighborhood construction, and fully totally different implementations of the equivalent neighborhood construction may end up in fully totally different outcomes. To make points additional tough, the effectivity of these fashions moreover differs primarily based on post-processing parameters, much like confidence thresholds, non-maximum suppression, and so forth.

Every writer casts their model as a result of the “best”, nevertheless I shortly realized that “best” relies upon upon context. There isn’t any purpose best model. There are two elementary requirements when deciding which face detection model is most relevant for the given context: accuracy and tempo.

No model combines extreme accuracy with extreme tempo; it’s a trade-off. We even have to take a look at metrics previous raw accuracy, on which most benchmarks are based (applicable guesses / full sample measurement), nevertheless raw accuracy won’t be the one metric to pay attention to. The ratio of false positives to true positives, and false negatives to true negatives, could be an very important consideration. In technical phrases, the trade-off is between precision (minimizing false positives) and recall (minimizing false negatives). This article discusses the difficulty in further depth.

There are a few current face detection datasets used for benchmarking, much like WIDER FACE, nevertheless I always want to see how the fashions will perform on my own information. So I randomly grabbed 1064 frames from my sample of TV reveals to verify the fashions ( ±3% margin of error). When manually annotating each image, I tried to select as many faces as attainable, along with faces that had been partially or almost completely occluded to offer the fashions an precise downside. On account of I’m in the end going to hold out facial recognition on the detected faces, I wanted to verify the boundaries of each model.

The images could be discovered to acquire with their annotations. I’ve moreover shared a Google Colab pocket guide to work along with the knowledge here.

It helps to group the numerous fashions into two camps; those that run on the GPU and people who run on the CPU. Normally, once you’ve obtained a CUDA-compatible GPU, it’s greatest to make use of a GPU-based model. I’ve an NVIDIA 1080 TI with 11GB of memory, which allows me to utilize among the many larger-scale fashions. Nonetheless, the size of my enterprise is giant (I’m talking a whole bunch of video recordsdata), so the lightning-fast CPU-based fashions intrigued me. There aren’t many CPU-based face detection fashions, so I decided to verify solely probably the most well-liked one: YuNet. Because of its tempo, YuNet varieties my baseline comparability. A GPU model needs to be significantly additional appropriate than its CPU counterpart to justify its slower processing tempo.

YuNet

YuNet was developed with effectivity in ideas with a model measurement that’s solely a fraction of the larger fashions. For instance, YuNet has solely 75,856 parameters compared with the 27,293,600 that RetinaFace boasts, which allows YuNet to run on “edge” computing items that aren’t extremely efficient enough for the larger fashions.

Code to implement the YuNet model shall be found on this repository. The only method to get YuNet up and dealing is through OpenCV.

cv2.FaceDetectorYN_create('./face_detection_yunet_2023mar.onnx',
"", 
(300, 300),
score_threshold=0.5) 

The pre-trained model is on the market on the OpenCV Zoo repository here. Merely make sure when cloning the repo to utilize Git LFS (I made that mistake at first). There’s a Google Colab file I wrote to show accessible here.

YuNet carried out a lot greater than I anticipated for a CPU model. It’s able to detect big faces and never utilizing a disadvantage nevertheless does wrestle a bit with smaller ones.

The accuracy improves enormously when limiting to the largest face throughout the image.

If effectivity is a serious concern, YuNet is an effective risk. It’s even fast enough for real-time functions, in distinction to the GPU selections accessible (on the very least with out some essential {{hardware}}).

Dlib

Dlib is a C++-implementation with a Python wrapper that maintains a stability between accuracy, effectivity, and luxury. Dlib shall be put in immediately by way of Python or accessed by way of the Face Recognition Python library. However, there’s a really strong trade-off between Dlib accuracy and effectivity primarily based totally on the upsampling parameter. When the number of situations to upsample is about to 0, the model is faster nevertheless a lot much less appropriate.

No Upsampling

Upsampling = 1

The accuracy of the Dlib model will enhance with further upsampling, nevertheless one thing elevated than upsampling=1 would set off my script to crash on account of it exceeded my GPU memory (which is 11GB by the best way by which).

Dlib’s accuracy was significantly disappointing relative to its (lack of) tempo. However, it was glorious at minimizing false positives, which is a priority of mine. Face detection is the first part of my facial recognition pipeline, so minimizing the number of false positives will help cut back errors downstream. To cut back the number of false positives even further, we’ll use Dlib’s confidence output to filter lower-confident samples.

There’s a large discrepancy in confidence between false and true positives, which we’ll use to filter out the earlier. Fairly than choose an arbitrary threshold, we’ll take a look on the distribution of confidence scores to select a additional actual one.

95% of the boldness values fall above 0.78, so excluding each half below that price reduces the number of false positives by half.

Whereas filtering by confidence reduces the number of false positives, it doesn’t improve the overall accuracy. I would consider using Dlib when minimizing the number of false positives is a serious concern. Nonetheless in every other case, Dlib doesn’t provide a giant enough improve in accuracy over YuNet to justify the so much elevated processing situations; on the very least for my features.

OpenCV DNN

The primary draw of OpenCV’s face detection model is its tempo. However, its accuracy left one factor to be desired. Whereas it’s extraordinarily fast when compared with the other GPU fashions, even its Prime 1 accuracy was hardly greater than YuNet’s complete accuracy. It’s unclear to me throughout which situation I would ever choose the OpenCV model for face detection, significantly since it could be tough to get working (you must assemble OpenCV from provide, which I’ve written about here).

Pytorch-MCNN

The MTCNN model moreover carried out pretty poorly. Although it was barely additional appropriate than the OpenCV model, it was pretty a bit slower. Since its accuracy was lower than YuNet, there was no compelling trigger to select MTCNN.

RetinaFace

RetinaFace has a reputation for being basically probably the most appropriate of open-source face detection fashions. The check out outcomes once more up that reputation.

Not solely was it basically probably the most appropriate model, nevertheless many of the “inaccuracies” weren’t, in actuality, exact errors. RetinaFace really examined the category of “false constructive” as a result of it picked up faces I hadn’t seen, hadn’t bothered to annotate on account of I believed them too troublesome, or hadn’t even considered a “face.”

It picked up a partial face in a mirror on this Seinfeld physique.

It managed to seek out faces in picture frames throughout the background of this Trendy Family.

And it’s so good at determining “faces,” that it finds non-human ones.

It was a pleasing shock learning that RetinaFace wasn’t all that gradual each. Whereas it wasn’t as fast as YuNet or OpenCV, it was much like MTCNN. Whereas it’s slower at lower resolutions than MTCNN, it scales comparatively properly and should course of elevated resolutions merely as shortly. And RetinaFace beat Dlib (on the very least when having to upscale). It’s so much slower than YuNet nevertheless is significantly additional appropriate.

Many of the “false positives” RetinaFace acknowledged shall be excluded by filtering out smaller faces. If we drop the underside quartile of faces, the false constructive charge drops drastically.

Whereas RetinaFace is extraordinarily appropriate, the errors do have a particular bias. Although RetinaFace identifies small faces with ease, it struggles with greater, partially occluded ones, which is apparent if we take a look at face measurement relative to accuracy.

This might probably be problematic for my features as a result of the measurement of a face in an image is strongly correlated to its significance. Subsequently, RetinaFace may miss an necessary circumstances, much like the occasion below.

Based totally on my exams (which I’d like to emphasize are normally not basically probably the most rigorous on this planet; so take them with a grain of salt), I would solely consider using each YuNet or RetinaFace, counting on whether or not or not tempo or accuracy was my main concern. It’s attainable I’d think about using Dlib if I fully wanted to attenuate false positives, nevertheless for my enterprise, it’s proper right down to YuNet or RetinaFace.

The GitHub repo used for this enterprise is on the market here.