Improve georeferencing (clipping) of uploaded files by using stadardized markers (e.g. ArUco Markers)

[matthiasschaub]

Initial pitch

Currently the most problems (detection but also performance) occur, due to a bad georeferencing or more precise, the clipping processes of the uploaded sketch.
At the moment there are 7 markers(globes) on the map with only 4 distinct once, also for right now the globs are only a support and not neccesary/mainly used for the clipping process, we need to change this!
There is a "standard" how to handle image detection/transformation with standardized markers. For example ArUco Markers (not sure what it stands for).
And opencv bring aruco marker generation and detection out-of-the-box.

• https://docs.opencv.org/4.x/d5/dae/tutorial_aruco_detection.html

We can complete get rid of the

• brisk
• flann
  and can use only the markers for clipping/transformation!

---

Update 1

PR #511 puts aruco marker on sketch map. This feature is optional and is implemented as a feature flag.

This PR does not detect markers on uploaded files. First experiments in detection of aruco markers were not very promising. The code is shared below.

def detect_aruco_markers(image: NDArray):
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    parameters = cv2.aruco.DetectorParameters()
    dictionary = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_50)
    detector = cv2.aruco.ArucoDetector(dictionary=dictionary, detectorParams=parameters)
    marker_corners, marker_ids, _ = detector.detectMarkers(gray)
    if marker_ids is None:
        # TODO: create custom exception
        raise AssertionError
    return marker_corners, marker_ids


@pytest.fixture
def sketch_map_photo():
    image = cv2.imread(str(FIXTURE_DIR / "upload-processing" / "sketch-map-photo.png"))
    return cv2.cvtColor(image, cv2.COLOR_RGB2BGR)


def test_detect_aruco_markers(sketch_map_photo):
    marker_corners, marker_ids = detect_aruco_markers(sketch_map_photo)
    cv2.aruco.drawDetectedMarkers(
        sketch_map_photo,
        marker_corners,
        marker_ids,
        borderColor=(0, 255, 0),  # red
    )
    # fmt: off
    options = (
            Options()
            .with_reporter(ImageReporter())
            .with_namer(PytestNamer())
    )
    # fmt: on
    verify_binary(
        cv2.imencode(".png", cv2.cvtColor(sketch_map_photo, cv2.COLOR_RGB2BGR))[
            1
        ].tobytes(),
        ".png",
        options=options,
    )

Using aruco markers instead of globes while keeping current workflow of clipping using BRISK might still be an improvement because currently globes are not unique.

Detection of aruco markers could be made more reliable by using aruco boards. This solution is more effort to implement because we can not use the regular grid boards provided by opencv. Instead, we would need to create our own boards based on paper size and different distances between markers in horizontal and vertical dimension. The code of first experiments with boards is shared below:

# NOTE: GridBoard can not be used,
# because it produces a regular grid with same distance
# between markers in horizontal and vertical dimension
def get_aruco_board(size, seperation):
    dictionary = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_50)
    board = cv2.aruco.GridBoard(
        size=(3, 3),
        markerLength=size,
        markerSeparation=seperation,
        dictionary=dictionary,
    )
    return board


def test_get_aruco_board():
    board = get_aruco_board(size=400, seperation=100)
    image = board.generateImage((1400, 1400))
    assert isinstance(image, np.ndarray)
    options = Options().with_reporter(NDArrayReporter()).with_namer(PytestNamer())
    # fmt: on
    verify_binary(
        serialize_ndarray(image),
        ".npy",
        options=options,
    )