Use a restricted quadratic, reflection and a fixed-point condition

The retained right-hand branch of the parabola has vertex $(-b/a,0)$ and is strictly increasing on $[-b/a,\infty)$. Every horizontal line cuts it at most once, so it is one-to-one and $f^{-1}$ exists.

The two graphs are reflections of one another in $y=x$. Since $f$ is increasing, their common points lie on that line: $f(x)=x$. Thus $(ax+b)^2=x$, or $a^2x^2+(2ab-1)x+b^2=0$. An intersection requires real roots, so $(2ab-1)^2-4a^2b^2=1-4ab\geq0$. As $b>0$, this yields $a\leq\dfrac1{4b}$.

The range $R_f=[0,\infty)$ lies inside the domain $D_f=[-b/a,\infty)$, since $a,b>0$. Therefore $ff$ is defined on the whole stated domain of $f$.

The equation becomes $\left[a(2a+b)^2+b\right]^2=b^2$. The quantity inside the square brackets is positive because $a,b>0$, so it must equal $b$. Therefore $a(2a+b)^2=0$, and since $a>0$ this would require $2a+b=0$, or $b=-2a$. This contradicts positivity of both constants. Thus there is no solution. Equivalently, the inner quantity is strictly larger than $b$, so its square is strictly larger than $b^2$.