First off, let’s label the unknowns:
Note that our objective function (namely, the area we are trying to maximize) can be expressed as a function of three variables:
A(x,y,z) = x\times z + y\times z.
Also, there are equations relating x, y, and z:
x^2 + z^2 = a^2 \text{ and } y^2 + z^2 = b^2
so that
x = \sqrt{a^2 - z^2} \text{ and } y = \sqrt{b^2-z^2}.
Thus, our area function can be expressed as a function of the single variable z:
A(z) = z\sqrt{a^2-z^2} + z\sqrt{b^2-z^2}.
So that’s the function we’ve got to maximize subject to 0 < z < a < b.
It doesn’t start out so bad:
\begin{aligned}
A'(z) &= \sqrt{a^2-z^2}-\frac{z^2}{\sqrt{a^2-z^2}} + \sqrt{b^2-z^2}-\frac{z^2}{\sqrt{b^2-z^2}} \\
&= \frac{a^2-2 z^2}{\sqrt{a^2-z^2}} - \frac{b^2-2 z^2}{\sqrt{b^2-z^2}} \stackrel{?}{=} 0,
\end{aligned}
which we can write
\frac{a^2-2 z^2}{\sqrt{a^2-z^2}} = \frac{b^2-2 z^2}{\sqrt{b^2-z^2}}.
Now, we square both sides to get rid of the square roots
\frac{(a^2-2 z^2)^2}{a^2-z^2} = \frac{(b^2-2 z^2)^2}{b^2-z^2},
which we can write as
(a^2-2 z^2)^2(b^2-z^2) - (b^2-2 z^2)^2(a^2-z^2) = 0.
Fortunately, there is a lot of cancellation when we expand the left and, in fact, we get
z^2b^4-z^2a^4 + a^4 b^2 - a^2 b^4 = 0.
Now, that’s pretty easy to solve for z and, in fact, we get
z = \sqrt{\left(a^2b^4 - a^4b^2\right)/\left(b^4 - a^4\right)}.
Finally, that tells us the value of z that we need and, looking back at the picture, we can figure out what the lengths of the diagonals are in terms of z.