Upvote:5
A 7-day visit is clearly not what a transit visa is for. It seems pretty obvious to me that your nephew needs to apply for a visitor visa.
There is no longer any such thing as a family visitor visa; it has been superseded by the UK Standard Visitor visa.
I don't know how your sister's visa status might affect the decision. I'd think that, as with most visa applications, it mostly comes down to convincing the visa officers that he will leave the UK when he says, and not cause trouble while he's there.