Yes, if you were to mix in some genes from a non-line bred variant, you'd need to spend some generations thereafter refining back to the specific line bred individuals you bad before.
For some species which grow faster, this isn't as bad, but as for myself, I've tried this before and found I do not have the patience to breed generations back to back to back over a period of years. I lose interest.
The very nature of line breeding leads to weak genetics and comes with the territory IMO.