Recover a dot product, projection and a parameterised area

$\begin{aligned}&|2\mathbf b-\mathbf a|^2\\ & =(2\mathbf b-\mathbf a)\cdot(2\mathbf b-\mathbf a)\\ & =4|\mathbf b|^2+|\mathbf a|^2-4\mathbf a\cdot\mathbf b\\ & =49.\end{aligned}$

Thus $4(3^2)+5^2-4\mathbf a\cdot\mathbf b=49$, so $4\mathbf a\cdot\mathbf b=12$ and $\mathbf a\cdot\mathbf b=3$.

The nonnegative projection length is

$|\mathbf a\cdot\widehat{\mathbf b}|=\frac{|\mathbf a\cdot\mathbf b|}{|\mathbf b|}=\frac33=1.$

The section formula gives

$OP=\frac{k\mathbf a+\mathbf b}{k+1}.$

Hence

$\begin{aligned}&\operatorname{area}(OAP)\\ & =\frac12|OP\times OA|\\ & =\frac1{2(k+1)}|k\mathbf a\times\mathbf a+\mathbf b\times\mathbf a|\\ & =\frac{|\mathbf a\times\mathbf b|}{2(k+1)}.\end{aligned}$

Here $\mathbf a\times\mathbf a=\mathbf0$ and reversing the cross product changes only its sign, not its magnitude.

Let θ be the angle between a,b. Then $\cos\theta=3/(5\cdot3)=1/5$. A right triangle with hypotenuse 5 and adjacent side 1 has opposite side $\sqrt{25-1}=2\sqrt6$, so $\sin\theta=2\sqrt6/5$. The source alternative is the identity $\sin\theta=\sqrt{1-\cos^2\theta}=\sqrt{1-1/25}=2\sqrt6/5$.

$\begin{aligned}&\operatorname{area}(OAP)\\ & =\frac{(5)(3)(2\sqrt6/5)}{2(k+1)}\\ & =\frac{3\sqrt6}{k+1}.\end{aligned}$

Positive k ensures the section is internal and k+1>0.