Turn a cross product with a unit direction into a distance

$OC=\mathbf a/3$, $OD=2\mathbf b/5$, so $OM=\frac12(OC+OD)=\mathbf a/6+\mathbf b/5$.

$\operatorname{area}(OMD)=\frac12|(\mathbf a/6+\mathbf b/5)\times(2\mathbf b/5)|=\frac12|(2/30)(\mathbf a\times\mathbf b)|=\frac1{30}|\mathbf a\times\mathbf b|$, since $\mathbf b\times\mathbf b=0$. Hence k=1/30.

$OM=\frac16(2,3,6)+\frac15(10,5,10)=(1/3,1/2,1)+(2,1,2)=(7/3,3/2,3)$. Since $|\mathbf b|=\sqrt{225}=15$,

$\begin{aligned}&OM\times\widehat{\mathbf b}\\ & =\frac1{15}[(7/3,3/2,3)\times(10,5,10)]\\ & =\frac1{15}(15-15,-(70/3-30),35/3-15)\\ & =\frac1{15}(0,20/3,-10/3)\\ & =\frac29(0,2,-1).\end{aligned}$

$|OM\times\widehat{\mathbf b}|$ is the perpendicular distance from M to the line OB. The second factor is unit length, so the cross-product magnitude is precisely the perpendicular component of OM. Its value, if desired, is $2\sqrt5/9$.