The more complex an organism, the more quickly inbreeding becomes problematic.
For africans, a few generations typically isn't an issue. If the fry aren't turning out properly, it's definitely a sign to get new blood in your group. I try when possible to get unrelated fish, for example I'd found some nice electric yellows, raised them up and got a nice group of females and then managed to find a few nice F1 males.
On the flip side I've got some F1 N. linni and intend on breeding those with one another once they are older (presuming I actually get some females this time - last time I had 5/6 as males).