Determines if a matrix is approximately orthogonal.

A matrix is orthogonal if it's square and real and its transpose is its inverse.

matrix The matrix to check.
rtol The per-matrix-entry relative tolerance on equality.
atol The per-matrix-entry absolute tolerance on equality.

Whether the matrix is orthogonal within the given tolerance.