This can be rephrased as $$(A+I)^{-1} = \frac{(1 + a_{01}^2 + a_{02}^2 + a_{12}^2)I - A + A^2}{|A+I|}$$ where $|A+I| = 1 + a_{01}^2 + a_{02}^2 + a_{12}^2$. For $4\times 4$ and $5\times 5$ it generalises respectively to $$-\frac{(1 + a_{01}^2 + a_{02}^2 + a_{03}^2 + a_{12}^2 + a_{13}^2 + a_{23}^2)(A - I) - A^2 + A^3}{|A+I|}$$ and $$\frac{(1 + a_{01}^2 + a_{02}^2 + a_{03}^2 + a_{04}^2 + a_{12}^2 + a_{13}^2 + a_{14}^2 + a_{23}^2 + a_{24}^2 + a_{34}^2)A(A-I) - A^3 + A^4}{|A+I|}$$ but the obvious extensions don't work for $6\times 6$, and Sage is running out of memory when trying to calculate an exact solution.

Thanks for this. I tested it out in Python and verified the 4x4 equation. But I think the 5x5 equation should have the identity matrix added to it: $$\frac{(1 + a_{01}^2 + a_{02}^2 + a_{03}^2 + a_{04}^2 + a_{12}^2 + a_{13}^2 + a_{14}^2 + a_{23}^2 + a_{24}^2 + a_{34}^2)A(A-I) - A^3 + A^4}{| A + I |} + I$$

Yes, you're right. I don't think I've kept all of my previous Sage code so I can't try to figure out where I made the mistake, but that does open up some other ideas for extensions to $6 \times 6$.