Post History

What you've done seems fine; you just need to continue to use the information that you have. In particular, that $s^2=10$. This is definitely an example where generalizing makes things more clear and simple. Abstracting from the details and removing unnecessary information makes the computation and laying out of what information you have easier.

The first thing to note, which is implicit in what you've written, is there is nothing special about one of the lines being an angle bisector. Given any two lines, we can always find a third such that one of the lines is an angle bisector of the others, so that is providing no additional information and certainly no necessary information. We can also readily see that the problem is translation invariant, i.e. solving the translated problem just gives a translated answer.

With these insights, the problem reduces to: Given two (linearly independent) vectors $\mathbf u$ and $\mathbf v$, find a vector $\mathbf s$ such that $\mathbf s = \alpha\mathbf u - \beta\mathbf v$, $\mathbf s\cdot \mathbf s = 10$, and $\mathbf s \cdot \mathbf v = 0$. To be clear, $\mathbf s$ is the vector from the point on the line induced by $\mathbf v$ to the point on the line induced by $\mathbf u$. The actual *point* that we want is (the origin plus) $\beta\mathbf v$ plus $\mathbf s$, i.e. it's (the origin plus) $\alpha\mathbf u$.

At this point, you simply start solving for $\alpha$ and $\beta$. Writing out all the information you have in a compact way by abstracting from the concrete problem and leveraging notation makes it much easier to formulate a plan. In this case, we want to start by seeing if we can eliminate $\beta$ as a free variable and $\mathbf s \cdot \mathbf v = 0$ is a simple way of doing that. It leads to the equation $\alpha\mathbf u\cdot \mathbf v = \beta \mathbf v^2$ which we quickly solve as $\beta = \alpha \mathbf u \cdot \mathbf v / \mathbf v^2$.

Using $\mathbf s \cdot \mathbf s = 10$ gives $$\begin{align}
\mathbf s \cdot (\alpha \mathbf u - \beta \mathbf v)
& = \mathbf s \cdot \alpha \mathbf u \\\\
& = \alpha^2 \mathbf u^2 - \beta\alpha\mathbf v\cdot \mathbf u \\\\
& = 10
\end{align}$$

Substituting the earlier expression for $\beta$ gives $\alpha^2(\mathbf u^2 - (\mathbf u \cdot \mathbf v)^2/\mathbf v^2) = 10$. This is easily solved for $\alpha$. If we chose, $\mathbf u$ and $\mathbf v$ to be normalized, i.e. of length $1$, then this simplifies to $\alpha^2 = 10/(1-\cos^2\theta)=10/\sin^2\theta$ where $\theta$ is the angle between the two lines. We can verify that this is the correct answer via trigonometry. (How?)

An alternative approach would be to recognize that the problem is also rotationally invariant, so you could actually rotate one of the lines to be the $x$-axis and the problem becomes simply what (rotated) point on the rotated bisector has $\sqrt{10}$ as its $y$-coordinate. Geometric algebra would allow us to express the rotations purely in terms of vectors, i.e. without introducing rotation matrices.

As a minor terminological nit that I often see, an expression like $\begin{pmatrix} 3\\0 \end{pmatrix} + t \begin{pmatrix} 1\\1\end{pmatrix}$ is not an equation – there is no equals sign. "Expression" or "term" would be more appropriate terminology.