I think this is looking pretty plausible, so running with it for a while I was wondering if there might be an obvious link between Robinson Crusoe and the name of the building 'Norfolk House'.
Well after copious amounts of dedicated research (hem, hem...) I'm happy to report that there is a link, but it's not all that obvious. Crusoe, it seems, was an unlucky mariner from the very start and after running away from home and boarding a ship from Hull managed to get as far as Cromer off of the Norfolk coast before sinking! He then walked to Yarmouth where he found some aid and assistance and wondered whether the fates were trying to tell him something... Very tenuous then, but luckily not the only one as it turns out.
The other link is to do with the name 'Robinson Crusoe' itself, which when you think about it is quite an odd name. The book itself explains its background
...my father being a foreigner of Bremen...settled first at Hull. He got a good estate by merchandise, and leaving off his trade, lived afterwards at York, from whence he had married my mother, whose relations were named Robinson, a very good family in that country, and from whom I was called Robinson Kreutznaer; but, by that usual corruption of words in England, we are now called - nay we call ourselves and write our name - Crusoe; and so my companions always called me.This is quite a bit of background colour in order to come up with such an unusual name - why not stick to something like Jack Jones or something? - but it looks as though the author Daniel Defoe may have picked the name up already before he even wrote the book. Defoe enjoyed and wrote about the town of King's Lynn in Norfolk, and as noted on the Literary Norfolk website 'In his A Tour Through the Whole Island of Great Britain Daniel Defoe was hugely impressed by King's Lynn preferring it, in many ways, to both Norwich and Yarmouth.' What Defoe might also have noticed would be the graves and gravestones in St Nichols' Chapel, Kings Lynn which happen to contain the earthly remains of several individuals by the name of Robinson Cruso!
Even if there's no real solid link between the figures of Crusoe and Friday (who incidentally, survived to appear in the sequel only to die half way through with three arrows in his chest) and the county of Norfolk, it was still very intriguing to spot what seemed to be a literary reference in a spot of doorway ornamentation. 'Norfolk House' is more than likely a bit of a red-herring - it could just be that the owner enjoyed a good read!