Post History

Firstly, I assume that your usage of "quadrilateral" implies planarity, and I will largely assume that the hexahedron is convex. Then the hexahedron can be represented as six vectors, being the (unnormalised) normals for the planes under the convention that the plane is given by $v \cdot N = 1$. This gives 18 degrees of freedom for the hexahedron, which is less than the 24 df of 8 vertices in general position because of the planarity constraints.

The smaller hexahedron $AIJKLMNO$ shares three of its planes with $ABCDEFGH$ so it has 9 df. The volume constraints on the subhexahedra give three independent constraints; I think the planarity constraints on $OKDH$, $OMFH$, $ONGH$ are also independent; so that leaves 3 df of flexibility.

To set up all of the volumes, the easiest approach is probably to assume that all the hexahexdra are quadrilateral; take the centroid of each one as the point of six pyramids with the hexahedron faces as their bases; if the point of a pyramid is $X$, the base is $ABCD$ in clockwise order, and $P$ is any point in the plane of the base, then the (signed) volume is $\frac16 (X-P) \cdot ((A-C) \times (B-D))$. For symmetry it's probably useful to take $P = \frac{A+B+C+D}{4}$. With the correct assignment of signs you get a quadratic programming setup to which you can add any objective function you want and tackle with standard techniques. It's probably worth trying to take $O = A + \frac{H-A}{\sqrt[3]4}$ as an initial guess to assign the signs.