Arulkumar03/Fox_Sheep_Detector_Computer_Vision_model

1# Copyright (c) Facebook, Inc. and its affiliates.2import math3from typing import List, Tuple4import torch5 6from detectron2.layers.rotated_boxes import pairwise_iou_rotated7 8from .boxes import Boxes9 10 11class RotatedBoxes(Boxes):12    """13    This structure stores a list of rotated boxes as a Nx5 torch.Tensor.14    It supports some common methods about boxes15    (`area`, `clip`, `nonempty`, etc),16    and also behaves like a Tensor17    (support indexing, `to(device)`, `.device`, and iteration over all boxes)18    """19 20    def __init__(self, tensor: torch.Tensor):21        """22        Args:23            tensor (Tensor[float]): a Nx5 matrix.  Each row is24                (x_center, y_center, width, height, angle),25                in which angle is represented in degrees.26                While there's no strict range restriction for it,27                the recommended principal range is between [-180, 180) degrees.28 29        Assume we have a horizontal box B = (x_center, y_center, width, height),30        where width is along the x-axis and height is along the y-axis.31        The rotated box B_rot (x_center, y_center, width, height, angle)32        can be seen as:33 34        1. When angle == 0:35           B_rot == B36        2. When angle > 0:37           B_rot is obtained by rotating B w.r.t its center by :math:`|angle|` degrees CCW;38        3. When angle < 0:39           B_rot is obtained by rotating B w.r.t its center by :math:`|angle|` degrees CW.40 41        Mathematically, since the right-handed coordinate system for image space42        is (y, x), where y is top->down and x is left->right, the 4 vertices of the43        rotated rectangle :math:`(yr_i, xr_i)` (i = 1, 2, 3, 4) can be obtained from44        the vertices of the horizontal rectangle :math:`(y_i, x_i)` (i = 1, 2, 3, 4)45        in the following way (:math:`\\theta = angle*\\pi/180` is the angle in radians,46        :math:`(y_c, x_c)` is the center of the rectangle):47 48        .. math::49 50            yr_i = \\cos(\\theta) (y_i - y_c) - \\sin(\\theta) (x_i - x_c) + y_c,51 52            xr_i = \\sin(\\theta) (y_i - y_c) + \\cos(\\theta) (x_i - x_c) + x_c,53 54        which is the standard rigid-body rotation transformation.55 56        Intuitively, the angle is57        (1) the rotation angle from y-axis in image space58        to the height vector (top->down in the box's local coordinate system)59        of the box in CCW, and60        (2) the rotation angle from x-axis in image space61        to the width vector (left->right in the box's local coordinate system)62        of the box in CCW.63 64        More intuitively, consider the following horizontal box ABCD represented65        in (x1, y1, x2, y2): (3, 2, 7, 4),66        covering the [3, 7] x [2, 4] region of the continuous coordinate system67        which looks like this:68 69        .. code:: none70 71            O--------> x72            |73            |  A---B74            |  |   |75            |  D---C76            |77            v y78 79        Note that each capital letter represents one 0-dimensional geometric point80        instead of a 'square pixel' here.81 82        In the example above, using (x, y) to represent a point we have:83 84        .. math::85 86            O = (0, 0), A = (3, 2), B = (7, 2), C = (7, 4), D = (3, 4)87 88        We name vector AB = vector DC as the width vector in box's local coordinate system, and89        vector AD = vector BC as the height vector in box's local coordinate system. Initially,90        when angle = 0 degree, they're aligned with the positive directions of x-axis and y-axis91        in the image space, respectively.92 93        For better illustration, we denote the center of the box as E,94 95        .. code:: none96 97            O--------> x98            |99            |  A---B100            |  | E |101            |  D---C102            |103            v y104 105        where the center E = ((3+7)/2, (2+4)/2) = (5, 3).106 107        Also,108 109        .. math::110 111            width = |AB| = |CD| = 7 - 3 = 4,112            height = |AD| = |BC| = 4 - 2 = 2.113 114        Therefore, the corresponding representation for the same shape in rotated box in115        (x_center, y_center, width, height, angle) format is:116 117        (5, 3, 4, 2, 0),118 119        Now, let's consider (5, 3, 4, 2, 90), which is rotated by 90 degrees120        CCW (counter-clockwise) by definition. It looks like this:121 122        .. code:: none123 124            O--------> x125            |   B-C126            |   | |127            |   |E|128            |   | |129            |   A-D130            v y131 132        The center E is still located at the same point (5, 3), while the vertices133        ABCD are rotated by 90 degrees CCW with regard to E:134        A = (4, 5), B = (4, 1), C = (6, 1), D = (6, 5)135 136        Here, 90 degrees can be seen as the CCW angle to rotate from y-axis to137        vector AD or vector BC (the top->down height vector in box's local coordinate system),138        or the CCW angle to rotate from x-axis to vector AB or vector DC (the left->right139        width vector in box's local coordinate system).140 141        .. math::142 143            width = |AB| = |CD| = 5 - 1 = 4,144            height = |AD| = |BC| = 6 - 4 = 2.145 146        Next, how about (5, 3, 4, 2, -90), which is rotated by 90 degrees CW (clockwise)147        by definition? It looks like this:148 149        .. code:: none150 151            O--------> x152            |   D-A153            |   | |154            |   |E|155            |   | |156            |   C-B157            v y158 159        The center E is still located at the same point (5, 3), while the vertices160        ABCD are rotated by 90 degrees CW with regard to E:161        A = (6, 1), B = (6, 5), C = (4, 5), D = (4, 1)162 163        .. math::164 165            width = |AB| = |CD| = 5 - 1 = 4,166            height = |AD| = |BC| = 6 - 4 = 2.167 168        This covers exactly the same region as (5, 3, 4, 2, 90) does, and their IoU169        will be 1. However, these two will generate different RoI Pooling results and170        should not be treated as an identical box.171 172        On the other hand, it's easy to see that (X, Y, W, H, A) is identical to173        (X, Y, W, H, A+360N), for any integer N. For example (5, 3, 4, 2, 270) would be174        identical to (5, 3, 4, 2, -90), because rotating the shape 270 degrees CCW is175        equivalent to rotating the same shape 90 degrees CW.176 177        We could rotate further to get (5, 3, 4, 2, 180), or (5, 3, 4, 2, -180):178 179        .. code:: none180 181            O--------> x182            |183            |  C---D184            |  | E |185            |  B---A186            |187            v y188 189        .. math::190 191            A = (7, 4), B = (3, 4), C = (3, 2), D = (7, 2),192 193            width = |AB| = |CD| = 7 - 3 = 4,194            height = |AD| = |BC| = 4 - 2 = 2.195 196        Finally, this is a very inaccurate (heavily quantized) illustration of197        how (5, 3, 4, 2, 60) looks like in case anyone wonders:198 199        .. code:: none200 201            O--------> x202            |     B\203            |    /  C204            |   /E /205            |  A  /206            |   `D207            v y208 209        It's still a rectangle with center of (5, 3), width of 4 and height of 2,210        but its angle (and thus orientation) is somewhere between211        (5, 3, 4, 2, 0) and (5, 3, 4, 2, 90).212        """213        device = tensor.device if isinstance(tensor, torch.Tensor) else torch.device("cpu")214        tensor = torch.as_tensor(tensor, dtype=torch.float32, device=device)215        if tensor.numel() == 0:216            # Use reshape, so we don't end up creating a new tensor that does not depend on217            # the inputs (and consequently confuses jit)218            tensor = tensor.reshape((0, 5)).to(dtype=torch.float32, device=device)219        assert tensor.dim() == 2 and tensor.size(-1) == 5, tensor.size()220 221        self.tensor = tensor222 223    def clone(self) -> "RotatedBoxes":224        """225        Clone the RotatedBoxes.226 227        Returns:228            RotatedBoxes229        """230        return RotatedBoxes(self.tensor.clone())231 232    def to(self, device: torch.device):233        # Boxes are assumed float32 and does not support to(dtype)234        return RotatedBoxes(self.tensor.to(device=device))235 236    def area(self) -> torch.Tensor:237        """238        Computes the area of all the boxes.239 240        Returns:241            torch.Tensor: a vector with areas of each box.242        """243        box = self.tensor244        area = box[:, 2] * box[:, 3]245        return area246 247    # Avoid in-place operations so that we can torchscript; NOTE: this creates a new tensor248    def normalize_angles(self) -> None:249        """250        Restrict angles to the range of [-180, 180) degrees251        """252        angle_tensor = (self.tensor[:, 4] + 180.0) % 360.0 - 180.0253        self.tensor = torch.cat((self.tensor[:, :4], angle_tensor[:, None]), dim=1)254 255    def clip(self, box_size: Tuple[int, int], clip_angle_threshold: float = 1.0) -> None:256        """257        Clip (in place) the boxes by limiting x coordinates to the range [0, width]258        and y coordinates to the range [0, height].259 260        For RRPN:261        Only clip boxes that are almost horizontal with a tolerance of262        clip_angle_threshold to maintain backward compatibility.263 264        Rotated boxes beyond this threshold are not clipped for two reasons:265 266        1. There are potentially multiple ways to clip a rotated box to make it267           fit within the image.268        2. It's tricky to make the entire rectangular box fit within the image269           and still be able to not leave out pixels of interest.270 271        Therefore we rely on ops like RoIAlignRotated to safely handle this.272 273        Args:274            box_size (height, width): The clipping box's size.275            clip_angle_threshold:276                Iff. abs(normalized(angle)) <= clip_angle_threshold (in degrees),277                we do the clipping as horizontal boxes.278        """279        h, w = box_size280 281        # normalize angles to be within (-180, 180] degrees282        self.normalize_angles()283 284        idx = torch.where(torch.abs(self.tensor[:, 4]) <= clip_angle_threshold)[0]285 286        # convert to (x1, y1, x2, y2)287        x1 = self.tensor[idx, 0] - self.tensor[idx, 2] / 2.0288        y1 = self.tensor[idx, 1] - self.tensor[idx, 3] / 2.0289        x2 = self.tensor[idx, 0] + self.tensor[idx, 2] / 2.0290        y2 = self.tensor[idx, 1] + self.tensor[idx, 3] / 2.0291 292        # clip293        x1.clamp_(min=0, max=w)294        y1.clamp_(min=0, max=h)295        x2.clamp_(min=0, max=w)296        y2.clamp_(min=0, max=h)297 298        # convert back to (xc, yc, w, h)299        self.tensor[idx, 0] = (x1 + x2) / 2.0300        self.tensor[idx, 1] = (y1 + y2) / 2.0301        # make sure widths and heights do not increase due to numerical errors302        self.tensor[idx, 2] = torch.min(self.tensor[idx, 2], x2 - x1)303        self.tensor[idx, 3] = torch.min(self.tensor[idx, 3], y2 - y1)304 305    def nonempty(self, threshold: float = 0.0) -> torch.Tensor:306        """307        Find boxes that are non-empty.308        A box is considered empty, if either of its side is no larger than threshold.309 310        Returns:311            Tensor: a binary vector which represents312            whether each box is empty (False) or non-empty (True).313        """314        box = self.tensor315        widths = box[:, 2]316        heights = box[:, 3]317        keep = (widths > threshold) & (heights > threshold)318        return keep319 320    def __getitem__(self, item) -> "RotatedBoxes":321        """322        Returns:323            RotatedBoxes: Create a new :class:`RotatedBoxes` by indexing.324 325        The following usage are allowed:326 327        1. `new_boxes = boxes[3]`: return a `RotatedBoxes` which contains only one box.328        2. `new_boxes = boxes[2:10]`: return a slice of boxes.329        3. `new_boxes = boxes[vector]`, where vector is a torch.ByteTensor330           with `length = len(boxes)`. Nonzero elements in the vector will be selected.331 332        Note that the returned RotatedBoxes might share storage with this RotatedBoxes,333        subject to Pytorch's indexing semantics.334        """335        if isinstance(item, int):336            return RotatedBoxes(self.tensor[item].view(1, -1))337        b = self.tensor[item]338        assert b.dim() == 2, "Indexing on RotatedBoxes with {} failed to return a matrix!".format(339            item340        )341        return RotatedBoxes(b)342 343    def __len__(self) -> int:344        return self.tensor.shape[0]345 346    def __repr__(self) -> str:347        return "RotatedBoxes(" + str(self.tensor) + ")"348 349    def inside_box(self, box_size: Tuple[int, int], boundary_threshold: int = 0) -> torch.Tensor:350        """351        Args:352            box_size (height, width): Size of the reference box covering353                [0, width] x [0, height]354            boundary_threshold (int): Boxes that extend beyond the reference box355                boundary by more than boundary_threshold are considered "outside".356 357        For RRPN, it might not be necessary to call this function since it's common358        for rotated box to extend to outside of the image boundaries359        (the clip function only clips the near-horizontal boxes)360 361        Returns:362            a binary vector, indicating whether each box is inside the reference box.363        """364        height, width = box_size365 366        cnt_x = self.tensor[..., 0]367        cnt_y = self.tensor[..., 1]368        half_w = self.tensor[..., 2] / 2.0369        half_h = self.tensor[..., 3] / 2.0370        a = self.tensor[..., 4]371        c = torch.abs(torch.cos(a * math.pi / 180.0))372        s = torch.abs(torch.sin(a * math.pi / 180.0))373        # This basically computes the horizontal bounding rectangle of the rotated box374        max_rect_dx = c * half_w + s * half_h375        max_rect_dy = c * half_h + s * half_w376 377        inds_inside = (378            (cnt_x - max_rect_dx >= -boundary_threshold)379            & (cnt_y - max_rect_dy >= -boundary_threshold)380            & (cnt_x + max_rect_dx < width + boundary_threshold)381            & (cnt_y + max_rect_dy < height + boundary_threshold)382        )383 384        return inds_inside385 386    def get_centers(self) -> torch.Tensor:387        """388        Returns:389            The box centers in a Nx2 array of (x, y).390        """391        return self.tensor[:, :2]392 393    def scale(self, scale_x: float, scale_y: float) -> None:394        """395        Scale the rotated box with horizontal and vertical scaling factors396        Note: when scale_factor_x != scale_factor_y,397        the rotated box does not preserve the rectangular shape when the angle398        is not a multiple of 90 degrees under resize transformation.399        Instead, the shape is a parallelogram (that has skew)400        Here we make an approximation by fitting a rotated rectangle to the parallelogram.401        """402        self.tensor[:, 0] *= scale_x403        self.tensor[:, 1] *= scale_y404        theta = self.tensor[:, 4] * math.pi / 180.0405        c = torch.cos(theta)406        s = torch.sin(theta)407 408        # In image space, y is top->down and x is left->right409        # Consider the local coordintate system for the rotated box,410        # where the box center is located at (0, 0), and the four vertices ABCD are411        # A(-w / 2, -h / 2), B(w / 2, -h / 2), C(w / 2, h / 2), D(-w / 2, h / 2)412        # the midpoint of the left edge AD of the rotated box E is:413        # E = (A+D)/2 = (-w / 2, 0)414        # the midpoint of the top edge AB of the rotated box F is:415        # F(0, -h / 2)416        # To get the old coordinates in the global system, apply the rotation transformation417        # (Note: the right-handed coordinate system for image space is yOx):418        # (old_x, old_y) = (s * y + c * x, c * y - s * x)419        # E(old) = (s * 0 + c * (-w/2), c * 0 - s * (-w/2)) = (-c * w / 2, s * w / 2)420        # F(old) = (s * (-h / 2) + c * 0, c * (-h / 2) - s * 0) = (-s * h / 2, -c * h / 2)421        # After applying the scaling factor (sfx, sfy):422        # E(new) = (-sfx * c * w / 2, sfy * s * w / 2)423        # F(new) = (-sfx * s * h / 2, -sfy * c * h / 2)424        # The new width after scaling tranformation becomes:425 426        # w(new) = |E(new) - O| * 2427        #        = sqrt[(sfx * c * w / 2)^2 + (sfy * s * w / 2)^2] * 2428        #        = sqrt[(sfx * c)^2 + (sfy * s)^2] * w429        # i.e., scale_factor_w = sqrt[(sfx * c)^2 + (sfy * s)^2]430        #431        # For example,432        # when angle = 0 or 180, |c| = 1, s = 0, scale_factor_w == scale_factor_x;433        # when |angle| = 90, c = 0, |s| = 1, scale_factor_w == scale_factor_y434        self.tensor[:, 2] *= torch.sqrt((scale_x * c) ** 2 + (scale_y * s) ** 2)435 436        # h(new) = |F(new) - O| * 2437        #        = sqrt[(sfx * s * h / 2)^2 + (sfy * c * h / 2)^2] * 2438        #        = sqrt[(sfx * s)^2 + (sfy * c)^2] * h439        # i.e., scale_factor_h = sqrt[(sfx * s)^2 + (sfy * c)^2]440        #441        # For example,442        # when angle = 0 or 180, |c| = 1, s = 0, scale_factor_h == scale_factor_y;443        # when |angle| = 90, c = 0, |s| = 1, scale_factor_h == scale_factor_x444        self.tensor[:, 3] *= torch.sqrt((scale_x * s) ** 2 + (scale_y * c) ** 2)445 446        # The angle is the rotation angle from y-axis in image space to the height447        # vector (top->down in the box's local coordinate system) of the box in CCW.448        #449        # angle(new) = angle_yOx(O - F(new))450        #            = angle_yOx( (sfx * s * h / 2, sfy * c * h / 2) )451        #            = atan2(sfx * s * h / 2, sfy * c * h / 2)452        #            = atan2(sfx * s, sfy * c)453        #454        # For example,455        # when sfx == sfy, angle(new) == atan2(s, c) == angle(old)456        self.tensor[:, 4] = torch.atan2(scale_x * s, scale_y * c) * 180 / math.pi457 458    @classmethod459    def cat(cls, boxes_list: List["RotatedBoxes"]) -> "RotatedBoxes":460        """461        Concatenates a list of RotatedBoxes into a single RotatedBoxes462 463        Arguments:464            boxes_list (list[RotatedBoxes])465 466        Returns:467            RotatedBoxes: the concatenated RotatedBoxes468        """469        assert isinstance(boxes_list, (list, tuple))470        if len(boxes_list) == 0:471            return cls(torch.empty(0))472        assert all([isinstance(box, RotatedBoxes) for box in boxes_list])473 474        # use torch.cat (v.s. layers.cat) so the returned boxes never share storage with input475        cat_boxes = cls(torch.cat([b.tensor for b in boxes_list], dim=0))476        return cat_boxes477 478    @property479    def device(self) -> torch.device:480        return self.tensor.device481 482    @torch.jit.unused483    def __iter__(self):484        """485        Yield a box as a Tensor of shape (5,) at a time.486        """487        yield from self.tensor488 489 490def pairwise_iou(boxes1: RotatedBoxes, boxes2: RotatedBoxes) -> None:491    """492    Given two lists of rotated boxes of size N and M,493    compute the IoU (intersection over union)494    between **all** N x M pairs of boxes.495    The box order must be (x_center, y_center, width, height, angle).496 497    Args:498        boxes1, boxes2 (RotatedBoxes):499            two `RotatedBoxes`. Contains N & M rotated boxes, respectively.500 501    Returns:502        Tensor: IoU, sized [N,M].503    """504 505    return pairwise_iou_rotated(boxes1.tensor, boxes2.tensor)506