Mask2Former, Dissected

In short. Image segmentation labels every pixel of an image. There are three versions of the task, semantic, instance and panoptic (§1). By late 2021, universal models, built to handle all three, did well on semantic and panoptic segmentation. But on instance segmentation, which separates individual objects, they still lost to specialized systems. MaskFormer, Mask2Former’s direct predecessor, trailed the best of them by over 9 AP, the standard instance score. Mask2Former [Cheng et al. 2022] kept MaskFormer’s core idea, one mask and one class per learned query, and rebuilt its decoder and its training recipe. It closed the gap, at 50.1 AP against 49.5 for the specialized Swin-HTC++, and reported the best results to date on the other two versions (§8).

How it works, in five steps.

1. The model keeps a fixed set of learned vectors called queries, 100 by default. Each query predicts one region of the image as a mask, and what that region is as a class (§2).
2. A query makes its mask by taking the dot product of its vector with the vector of every pixel. A high value means the pixel belongs to the query’s region (§3).
3. In training, each real object is paired with exactly one query, which learns to predict it. A pair’s cost measures how badly the query’s class and mask fit the object, and the pairing with the lowest total cost wins. Unpaired queries learn to predict “no object” (§4).
4. Nine decoder layers refine the queries. In each layer, a query reads image features only inside the mask it predicted at the previous layer, so the background can no longer drown out its object. This is masked attention, the paper’s main idea (§5).
5. Two more ideas make this affordable. The layers take turns reading coarse and fine feature maps, and the mask losses are computed on 12,544 sampled points instead of every pixel, which cuts training memory from 18 GB to 6 GB per image (§6).

Steps 1 to 3 come from MaskFormer, and steps 4 and 5 are new. Together they form a loop: every layer’s mask is trained against the true mask, and each trained mask decides where the next layer looks.

How to read this.

• In five minutes: the five steps and the picture above, then Fig. 2, Fig. 3, Fig. 4, Fig. 5 and Fig. 6, which show the steps in order.
• If you know DETR and the segmentation metrics: start at §3. Mask2Former’s own changes are §5 to §7, and §7 ends with what each one is worth.
• If any of this is new: start at §0, which builds every idea the rest uses. If attention and losses are familiar, start at §1.

Proofs and longer derivations are folded, and the main text never depends on them. In the figures, gold marks what the model predicts, green marks ground truth, and slate marks image features.

0. The background you need

Mask2Former is built from a few standard ideas, one per paragraph below. Skip any you already know.

Feature maps. A neural network called a backbone turns the image into a smaller grid, where each cell holds a vector that summarizes a patch of the image. That grid is a feature map, and its stride is the shrink factor. At stride 32, a 1024 by 1024 image becomes a 32 by 32 grid, so each cell stands for a 32 by 32 patch. Small objects lose detail at that scale, which is why Mask2Former also reads finer maps (§6).

Embeddings and the dot product. An embedding is a learned vector whose direction carries meaning. Related inputs tend to point in similar directions. The dot product measures how much two vectors agree: multiply them entry by entry, then add. For example, $(2, 1)\cdot(1, 2) = 4$, while $(2, 1)\cdot(-1, 2) = 0$. In general, for vectors $u$ and $v$ with angle $\theta$ between them,

where $u^\top$ is $u$ written as a row and $\|u\|$ is its length. The result is positive when the vectors point the same way, zero when they are perpendicular, and negative when they point apart. Mask2Former makes every mask this way (§3).

Softmax. Softmax turns a list of scores into weights that are positive and add up to 1. Take the scores 2.0, −0.6, −2.2 and 1.1. Exponentiating gives 7.39, 0.55, 0.11 and 3.00. Dividing each by their sum, 11.05, gives the weights 0.67, 0.05, 0.01 and 0.27. The largest score takes most of the weight, but no weight is exactly zero. In general, for scores $z_1, \dots, z_n$, which are also called logits,

The only way to give an option a weight of exactly 0 is a score of $-\infty$, because $e^{-\infty} = 0$. The other weights still add up to 1. Masked attention in §5 is built on this.

Attention. Attention [Vaswani et al. 2017] lets one vector, the query, collect information from many others. Each image location carries two vectors, both computed from its features by learned linear maps. Its key is compared with the query, and its value is what it hands over. The dot product of the query with each key gives a score, softmax turns the scores into weights, and the output is the values mixed by those weights. Fig. 0 runs this on four locations of a photo, with the scores of the softmax example.

Transformer decoder layers. A decoder layer updates every query’s vector $x$ three times, each time adding the result to $x$, which is called a residual connection. Leaving out normalization and projections, Mask2Former’s layer runs

The first update is cross-attention, where the query reads the image. The second is self-attention, where the queries read each other, so they can coordinate, for example to avoid describing the same object twice. The third is a small feed-forward network (FFN) applied to each query on its own. Mask2Former stacks nine such layers.

Losses and gradients. A loss $\mathcal{L}$ is one number that measures how wrong the output is. Training nudges every weight against the gradient $\nabla_w \mathcal{L}$, the direction in weight space that raises the loss fastest. Mask2Former’s loss is defined in §4.

1. Three ways to segment an image

Segmentation groups pixels and labels the groups. Datasets split categories into things, countable objects such as people and cars, and stuff, amorphous regions such as road and sky [Kirillov et al. 2019]. Fig. 1 shows one street scene segmented all three ways.

Semantic segmentation gives every pixel a class, so two touching cars become one car region. Instance segmentation returns one mask per thing, each with a confidence score, and ignores stuff. Panoptic segmentation [Kirillov et al. 2019] does both: every pixel gets a class, and pixels of things also get an instance id.

How the tasks are scored. Each task has its own score, from 0 to 100, where higher is better: mIoU for semantic, AP for instance and PQ for panoptic segmentation. All three rest on intersection over union (IoU), the area two regions share divided by their combined area, which is 1 for identical regions and 0 for regions that do not touch.

The results in this article use two datasets: COCO [Lin et al. 2014] for instance and panoptic segmentation, and ADE20K [Zhou et al. 2017] for semantic segmentation.

The exact definitions are folded below.

2. Mask classification: predict regions, not pixels

A model can produce a segmentation in two ways. It can label every pixel, or it can predict a set of regions and label each region. Mask2Former does the second.

Labeling pixels. A per-pixel classifier asks the same question at every pixel: which class is this? The fully convolutional network (FCN) [Long et al. 2015] made this the standard approach to semantic segmentation. It is trained with cross-entropy at every pixel, the negative log of the probability it gives the correct class.

The catch is object identity. A per-pixel classifier has no variable for which object a pixel belongs to, so it cannot separate two touching cars. Instance systems added extra machinery for that: region proposals, anchor boxes, grouping rules, or non-maximum suppression (NMS), a step that deletes overlapping duplicate detections.

Predicting regions. Mask classification predicts regions instead. The model keeps $N$ learned query vectors. Each query predicts one mask, which says where its segment is, and one class, which says what the segment is:

where the mask $m_i$ gives every pixel a probability of belonging to segment $i$, $K$ is the number of classes, and $\varnothing$ is an extra “no object” class for queries that find nothing. The mask and the class come from the same query, so they describe the same segment. Object identity comes for free, because two cars are two queries. The design comes from DETR [Carion et al. 2020], a detector built on learned queries, by way of MaskFormer [Cheng et al. 2021], Mask2Former’s direct predecessor.

Fig. 2 shows one such query on a real photo.

The same format covers all three tasks. Semantic targets contain one mask per class present in the image, instance targets one mask per object, and panoptic targets one mask per thing or stuff segment.

Mask2Former’s thesis is that this format was right, and that MaskFormer’s decoder and training recipe held it back.

3. How a query becomes a mask

Mask2Former keeps MaskFormer’s three parts, drawn in the overview figure at the top. Each part hands its output to the next.

1. A backbone, either a ResNet [He et al. 2016] or a Swin Transformer [Liu et al. 2021], turns the image into feature maps.
2. A pixel decoder refines them into feature maps at three resolutions, 1/32, 1/16 and 1/8 of the input. It also produces a pixel embedding map $\mathcal{E}_{\text{pixel}}$ at 1/4 resolution, with one vector of $C = 256$ numbers per location.
3. A Transformer decoder updates the $N$ query vectors in nine layers, each reading one of the three feature maps (§5, §6).

After every decoder layer, two small heads read each query’s vector $q_i$. The class head is a linear layer with a softmax, which gives one probability per class plus one for no-object. The mask head is a small multilayer perceptron (MLP) that turns the query into a mask embedding, a vector the same length as each pixel embedding. The mask is the sigmoid of their dot product at every pixel $x$:

where $\sigma(z) = 1/(1+e^{-z})$ turns the dot product, called the mask logit, into a probability. A pixel whose embedding points the same way as the query’s mask embedding gets a high probability. Every part, from the backbone to the heads, is trained together on the loss of §4, so the pixels of an object learn embeddings that line up with its query. Fig. 3 shows this on the real model.

Masks are computed at 1/4 of the input resolution, 256 by 256 for a 1024 by 1024 image, and then upsampled. Because the heads run after every layer, each query makes ten predictions: one from the learnable initial queries and one after each of the nine layers. Training supervises all ten (§4.2), and each of the first nine also decides where the next layer may look (§5).

From predictions to the output. At inference, the final $N$ predictions become the answer each task needs. Panoptic output gives each pixel to the confident query with the highest class probability times mask probability. Instance output ranks every query and class pair by its probability and keeps the top 100 as detections. Semantic output weights each mask by its class probabilities and labels every pixel with the class that collects the most weight. The exact rules are folded below.

4. Training: one query per object

Training has to decide which query should predict which object, and nothing in the output says so. The $N$ predictions form a set. Query 7 is not meant to predict object 7, and shuffling the queries only reorders the same predictions. So the loss works in two steps: matching pairs each real object with one query, and the loss then scores each pair. Mask2Former inherits this from DETR and MaskFormer. If you know DETR’s matching, skip to the last paragraph of §4.2, on deep supervision.

4.1 Matching predictions to objects

Letting each prediction pick the target it fits best does not work: two queries can pick the same object and leave another with none. Matching instead gives each target exactly one prediction, choosing the pairing with the lowest total cost. Every prediction left over learns to say no-object. That includes a second query on an object that is already taken, so the model learns not to predict duplicates, and inference needs no non-maximum suppression.

Try a small case first. Fig. 4 holds the costs of pairing three predictions with three targets. Before it plays, find the cheapest way to give each target its own prediction.

The cost. In the model, the cost of pairing prediction $i$ with target $j$ has three parts. A class term rewards a prediction that already gives the target’s class $c^{\text{gt}}_j$ a high probability. Two mask terms measure how far the predicted mask $m_i$ is from the target mask $m^{\text{gt}}_j$: binary cross-entropy (BCE), which scores each pixel on its own, and Dice, which scores the overlap of the whole region (both in §4.2). In full,

where the indicator $\mathbb{1}[\cdot]$ is 1 when its condition holds and 0 otherwise, so pairing a query with no object costs nothing. The weights $\lambda_{\text{cls}}$, $\lambda_{\text{ce}}$ and $\lambda_{\text{dice}}$ are 2, 5 and 5, and the minus sign makes a confident correct class lower the cost.

The assignment. With 100 queries there are far too many pairings to try them all. The Hungarian algorithm [Kuhn 1955, Munkres 1957] finds the cheapest one directly, in $O(N^3)$ time in its refined form. In code it is one call to scipy.optimize.linear_sum_assignment. The two folds below show why its answer is optimal and run it by hand on the matrix of Fig. 4.

4.2 The training loss

Each output is trained with three terms: a class loss on all $N$ queries, and two mask losses on the queries matched to real objects. The loss weights them as

The class loss $\mathcal{L}_{\text{cls}}$ is cross-entropy, with no-object targets weighted 0.1, because most queries are unmatched: with 100 queries and 5 objects, 95 learn no-object. BCE, $\mathcal{L}_{\text{ce}}$, scores each mask point on its own and keeps pushing a point that is confidently wrong. Dice, $\mathcal{L}_{\text{dice}}$ [Milletari et al. 2016], scores the overlap of the whole region, from 0 for a perfect match to almost 1 for none, so a small object counts as much as a large one at the same fraction of overlap. Its push on a confidently wrong point fades, which is why BCE is kept beside it. Both are evaluated at sampled points (§6.2).

Deep supervision. All ten outputs of §3 are trained with this loss, through the same shared heads. The matching is recomputed for each output, so every intermediate prediction trains toward its own assignment. This matters more than usual here: the intermediate masks become the attention masks of §5, so these losses also steer where each layer looks.

5. Masked attention: each query reads only its own region

In a standard decoder layer, every query reads the whole image through cross-attention. A query meant to describe the dog still spends attention on every patch of blanket. Global attention can learn where an object is, but the authors argue that learning this is slow, as earlier DETR studies had found [Gao et al. 2021, Sun et al. 2021]. Masked attention removes the problem by construction: each query reads only inside the mask it predicted at the previous layer.

5.1 Why global attention spreads thin

A small calculation shows how the background wins. Suppose the object covers 2 percent of the image, and every object location scores 2 higher than every background location. There are 49 background locations for each object location, and each of them gets $e^{-2} \approx 0.135$ of an object location’s weight. The object’s share of the attention is then only

The background wins by numbers, not by score. Real scores vary, so this is only a toy, but it shows why reading only the object’s region can make learning easier.

5.2 The masked update

Masked attention keeps the attention step of Fig. 0 and adds one term to the scores. Before reading on, take the four locations of Fig. 0 and suppose the previous mask keeps locations 1 and 4. What should happen to the weights of locations 2 and 3, and to the other two? Fig. 5 plays it out.

The gain is small here because locations 2 and 3 held little weight. In the toy image of §5.1, where the background held 87 percent of the attention, a gate that keeps only the object lifts the object’s share from 0.13 to 1.

In symbols, masked attention adds a mask $\mathcal{M}$ to the scores before the softmax. For query $i$, with the softmax taken over image locations $x$,

The previous mask is the one the query predicted at the layer before, resized to the feature map this layer reads and thresholded at probability 0.5. The first layer uses the mask predicted from the learned starting queries (§7). Self-attention, the next step in the layer, still carries information between queries, so a query that reads only its own region can still learn from the others.

Does it help? The paper swaps out one part at a time to find out. With a ResNet-50 (R50) backbone on COCO, plain cross-attention reaches 37.8 AP and masked attention 43.7. Even mask pooling, which simply averages the features inside the mask, reaches 43.1. So on COCO, most of the gain comes from limiting where a query reads, and masked attention adds a learned weighting inside the region.

The gate is a hard threshold, so no gradient passes through it. The intermediate masks learn from deep supervision instead (§4.2), which compares each intermediate prediction directly with the target mask.

attn_mask = (mask_logits.sigmoid() < 0.5).flatten(2)   # True = forbidden
attn_mask[attn_mask.all(dim=-1)] = False               # Fall back to global attention.

5.3 The effect, measured and seen

The paper also measures how much of each query’s attention lands inside its own object, the true segment it was matched to. With standard cross-attention the share is about 0.20. With masked attention it is 0.59. It stays below 1 partly because the gate is the previous predicted mask, at a coarse resolution, not the true one.

Fig. 6 shows the mechanism in the released Swin-L checkpoint. The query’s class settles early: it says dog with probability 0.98 after layer 2, and at least 0.998 from layer 3 on.

6. Making it affordable

Masked attention is the main idea. Two more ideas make it affordable: each layer reads only one feature resolution, and the mask losses are computed on sampled points instead of every pixel.

6.1 One resolution per layer

Small objects need fine feature maps, but attention over a fine map is expensive. Its cost grows with the number of locations. For a 1024 by 1024 input, the 1/32, 1/16 and 1/8 maps hold 1,024, 4,096 and 16,384 locations, so the finest map costs about sixteen times more than the coarsest.

Mask2Former reads one scale per layer. Layers 1, 2 and 3 read the 1/32, 1/16 and 1/8 maps, and the cycle repeats twice more, as Fig. 6 shows on the dog query. This round-robin reaches 43.7 AP at 226 GFLOPs, billions of operations per image. Feeding all three scales to every layer costs 21 more GFLOPs and is no better overall: 0.3 AP higher, the same PQ, and 0.9 mIoU lower.

The pixel decoder builds these maps by letting every location mix in information from all three scales. It uses multi-scale deformable attention (MSDeformAttn) from Deformable DETR [Zhu et al. 2021], in which each location reads a few learned sample points per scale rather than every location.

6.2 Mask losses on 12,544 points

For every output it trains, MaskFormer compared whole masks pixel by pixel twice: once for every prediction-target pair to build the matching costs, and again for the matched pairs to compute the loss. That limited training to one image per 32 GB GPU. Following PointRend [Kirillov et al. 2020], Mask2Former evaluates these terms at only $K_{\text{pt}} = 12{,}544 = 112^2$ sampled points per mask.

Matching uses shared uniform points. One set of random points, spread uniformly, is shared by every prediction and target mask in the image, which makes the cost matrix cheaper to build.

Training uses uncertain points. For each matched mask, the PointRend sampler draws $3K_{\text{pt}} = 37{,}632$ uniform candidates [Kirillov et al. 2020]. It keeps the 9,408 where the prediction is least sure, the logits closest to zero, then adds 3,136 fresh uniform points. The uncertainty score depends on the prediction, not the ground truth. It is also relative: the sampler always keeps the least-sure quarter of its candidates, so where the unsure band is thin, it also takes points that are only somewhat unsure. Fig. 7 runs both rules on the dog prediction.

Does sampling hurt accuracy? The paper tried each use on its own:

Rows 1 and 2 show that training on points instead of whole masks cuts memory per image from 18 GB to 6 GB at equal accuracy. Rows 1 and 3 show that matching on points adds 2.1 AP.

7. Smaller changes, and what each part is worth

Three smaller changes to the decoder layer cost no extra computation. Together they improve AP by about 1.4, PQ by 1.1, and mIoU by 0.9.

Masked attention comes first. DETR’s layer starts with self-attention. Mask2Former’s starts with masked attention, so each query has image context before the queries interact.

Query features are learnable and directly supervised. DETR starts every query’s features at zero and tells the queries apart only by a learned vector per query, its positional embedding, added before every attention step. Mask2Former keeps that embedding but also learns the starting features $\mathbf{X}_0$, and supervises their masks before the decoder runs, which gives the first layer a useful gate. Learning $\mathbf{X}_0$ raises mIoU, and supervising it raises AP and PQ. The paper compares these first predictions to region proposals [Ren et al. 2015] that the decoder refines.

Dropout is removed. Dropout randomly zeroes activations during training. Taking it out of the decoder improves AP and PQ.

A new training recipe. On COCO, Mask2Former trains for 50 epochs (passes over the training data) at batch 16, where MaskFormer used 300 at batch 64, with large-scale jittering (random rescaling between 0.1 and 2 times) and the loss weights of §4.2. The full comparison is folded below.

What each part is worth. The paper takes each part out of the full model, one at a time, and measures the drop. All of these runs use the R50 backbone.

*Measured with point-sampled training loss fixed. The full two-by-two comparison is in §6.2.

The effects overlap, so the rows do not add up. Taken out one at a time, the three layer changes cost 2.0 AP in total, but together they are worth 1.4.

The paper also builds Mask2Former from MaskFormer one step at a time on COCO. Retraining MaskFormer with Mask2Former’s recipe raises AP from 34.0 to 37.8. Replacing the decoder raises it to 41.5. Replacing MaskFormer’s feature pyramid network (FPN) pixel decoder [Lin et al. 2017b] with MSDeformAttn reaches 43.7. The steps interact, so these gains depend on the order and are not fixed shares of the total.

8. Results and limitations

8.1 Results

The paper’s largest models use a Swin-L backbone. The COCO models also use 200 queries and train for 100 epochs, twice the default.

The ADE20K result uses inference at several image sizes and a FaPN pixel decoder [Huang et al. 2021] instead of the default. The COCO results use one size. On COCO’s test set, Mask2Former’s 50.5 AP beats Swin-HTC++ at one size, 50.2, but not at several sizes, 51.1.

With a ResNet-50 backbone, Mask2Former reaches 51.9 PQ after 50 epochs, where MaskFormer reaches 46.5 after 300. MaskFormer’s epochs used batches four times larger, and the paper finds it does about as well in 75 epochs at Mask2Former’s batch of 16. Augmentation and losses differ too, so this is not a controlled test of training speed.

The gain is not just scale. Swin-T Mask2Former reaches 53.2 PQ at 232 GFLOPs, above Swin-L MaskFormer’s 52.7 at 792 GFLOPs.

8.2 Limitations

Universal means one architecture, not one checkpoint. A model trained on panoptic labels usually does slightly worse than the same model trained for one task: 41.7 against 43.7 AP on COCO instances, and 46.1 against 47.2 mIoU on ADE20K. Panoptic training does not yet replace task-specific training.

Small objects remain weaker. On COCO’s validation set, Mask2Former’s AP on small objects, under $32^2$ pixels, is 29.9, below Swin-HTC++ at 31.0, although it leads overall.

It is slower, and the query count depends on the task. Mask2Former runs at 8.6 frames per second against MaskFormer’s 17.6 for the ResNet-50 panoptic models. In the paper’s tests, 100 queries work best for instance and semantic segmentation, while 200 help panoptic segmentation slightly.

9. Conclusion

Mask2Former works because its changes form a feedback loop. Matching assigns one target to each query. Deep supervision trains every intermediate mask. Each mask restricts the next cross-attention step, and point sampling makes this repeated supervision affordable.

Later models build on the same design: Mask2Former for video [Cheng et al. 2021b], OneFormer [Jain et al. 2023], which trains one model for all three tasks, and Mask DINO [Li et al. 2023], which adds DETR-style detection. Mask2Former’s lasting contribution is this query-to-mask refinement loop, more than any one benchmark result.

Appendix A. Implementation notes

The reference implementation is facebookresearch/Mask2Former, built on Detectron2 [Wu et al. 2019]. Hugging Face provides it as Mask2FormerForUniversalSegmentation.

def decoder_layer(x, feats_l, size_l, pos_l, lvl_emb, query_pos, prev_mask_logits, pixel_embeddings, num_heads):
    # x: (N,B,C) · feats_l: (H_l*W_l,B,C) · size_l: (H_l,W_l)
    m = F.interpolate(prev_mask_logits, size=size_l, mode="bilinear", align_corners=False)
    attn_mask = (m.sigmoid() < 0.5).flatten(2)  # (B,N,H_l*W_l)
    attn_mask = attn_mask[:, None].repeat(1, num_heads, 1, 1).flatten(0, 1).detach()  # (B*num_heads,N,H_l*W_l)
    attn_mask[attn_mask.all(dim=-1)] = False  # Prevent all-masked softmax.

    # The scale embedding is part of both the key and value memory.
    feats_l = feats_l + lvl_emb
    x = norm1(x + cross_attn(q=with_pos(x, query_pos), k=with_pos(feats_l, pos_l), v=feats_l, attn_mask=attn_mask))
    x = norm2(x + self_attn(q=with_pos(x, query_pos), k=with_pos(x, query_pos), v=x))
    x = norm3(x + ffn(x))

    h = decoder_norm(x).transpose(0, 1)
    cls_logits = class_head(h)
    mask_logits = einsum("bnc,bchw->bnhw", mask_head(h), pixel_embeddings)
    return x, cls_logits, mask_logits

References

1. Cheng, Misra, Schwing, Kirillov, Girdhar. “Masked-attention Mask Transformer for Universal Image Segmentation.” 2022. arXiv:2112.01527 · Code
2. Cheng, Schwing, Kirillov. “Per-Pixel Classification is Not All You Need for Semantic Segmentation.” 2021. arXiv:2107.06278
3. Carion, Massa, Synnaeve, Usunier, Kirillov, Zagoruyko. “End-to-End Object Detection with Transformers.” 2020. arXiv:2005.12872
4. Kirillov, He, Girshick, Rother, Dollár. “Panoptic Segmentation.” 2019. arXiv:1801.00868
5. Kirillov, Wu, He, Girshick. “PointRend: Image Segmentation as Rendering.” 2020. arXiv:1912.08193
6. Zhu, Su, Lu, Li, Wang, Dai. “Deformable DETR: Deformable Transformers for End-to-End Object Detection.” 2021. arXiv:2010.04159
7. Zhang, Pang, Chen, Loy. “K-Net: Towards Unified Image Segmentation.” 2021. arXiv:2106.14855
8. Wang, Zhu, Adam, Yuille, Chen. “MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers.” 2021. arXiv:2012.00759
9. He, Gkioxari, Dollár, Girshick. “Mask R-CNN.” 2017. arXiv:1703.06870
10. Long, Shelhamer, Darrell. “Fully Convolutional Networks for Semantic Segmentation.” 2015. arXiv:1411.4038
11. Vaswani et al. “Attention Is All You Need.” 2017. arXiv:1706.03762
12. Kuhn. “The Hungarian Method for the Assignment Problem.” 1955. doi:10.1002/nav.3800020109
13. Munkres. “Algorithms for the Assignment and Transportation Problems.” 1957. doi:10.1137/0105003
14. Milletari, Navab, Ahmadi. “V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation.” 2016. arXiv:1606.04797
15. Lin, Goyal, Girshick, He, Dollár. “Focal Loss for Dense Object Detection.” 2017. arXiv:1708.02002
16. Lin, Dollár, Girshick, He, Hariharan, Belongie. “Feature Pyramid Networks for Object Detection.” 2017. arXiv:1612.03144
17. Gao, Zheng, Wang, Dai, Li. “Fast Convergence of DETR with Spatially Modulated Co-Attention.” 2021. arXiv:2101.07448
18. Sun, Cao, Yang, Kitani. “Rethinking Transformer-based Set Prediction for Object Detection.” 2021. arXiv:2011.10881
19. Liu et al. “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.” 2021. arXiv:2103.14030
20. He, Zhang, Ren, Sun. “Deep Residual Learning for Image Recognition.” 2016. arXiv:1512.03385
21. Chen et al. “Hybrid Task Cascade for Instance Segmentation.” 2019. arXiv:1901.07518
22. Ren, He, Girshick, Sun. “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.” 2015. arXiv:1506.01497
23. Huang, Lu, Cheng, He. “FaPN: Feature-Aligned Pyramid Network for Dense Image Prediction.” 2021. arXiv:2108.07058
24. Loshchilov, Hutter. “Decoupled Weight Decay Regularization.” 2019. arXiv:1711.05101
25. Ghiasi et al. “Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation.” 2021. arXiv:2012.07177
26. Bao, Dong, Wei. “BEiT: BERT Pre-Training of Image Transformers.” 2022. arXiv:2106.08254
27. Everingham et al. “The PASCAL Visual Object Classes Challenge: A Retrospective.” 2015. doi:10.1007/s11263-014-0733-5
28. Lin et al. “Microsoft COCO: Common Objects in Context.” 2014. arXiv:1405.0312
29. Zhou et al. “Scene Parsing through ADE20K Dataset.” 2017. Project page
30. Cheng et al. “Mask2Former for Video Instance Segmentation.” 2021. arXiv:2112.10764
31. Jain et al. “OneFormer: One Transformer to Rule Universal Image Segmentation.” 2023. arXiv:2211.06220
32. Li et al. “Mask DINO: Towards a Unified Transformer-based Framework for Object Detection and Segmentation.” 2023. arXiv:2206.02777
33. Wu, Kirillov, Massa, Lo, Girshick. “Detectron2.” 2019. Code

Citation

Massih, Peter. “Mask2Former, Dissected.” peteramassih.com, Jul 2026. https://peteramassih.com/writing/mask2former/.

@article{massih2026mask2former,
  title   = {Mask2Former, Dissected},
  author  = {Massih, Peter},
  journal = {peteramassih.com},
  year    = {2026},
  month   = {July},
  url     = {https://peteramassih.com/writing/mask2former/}
}