English Heritage is urging action be taken over a decaying East Cowes building, which it says would probably feature on a new 'at risk' register.
The Grade II listed building has suffered from years of indecision and neglect, according to the national body.
English Heritage said that details of the 1903-built memorial hospital's re-use as a residential property had been agreed, but it has been a 'hard slog' and local people were understandably concerned that nothing appears to be happening.
The former hospital building is an example of a Grade II building that English Heritage would probably feature on a new 'Heritage at Risk Register'. Grade II buildings are not currently covered by the organisation, which hopes to launch a pilot to protect up to 5,000 additional buildings at risk on the South East Register.
Dr Andy Brown, Planning Director for the South East, said: "We need to find some partners to work with us on the ground, so that we can complete the catalogue so that we really know what the whole picture is and so that we can prioritise our work in helping local communities to address the sorts of buildings that they cherish in their local places."
And he urged local action to be taken over the Frank James Hospital site. Dr Brown added: "Sometimes the local council has powers to make sure that these buildings are looked after properly. And sometimes we need to remind them of their powers and sometimes they need encouragement as well. So it's not all about spending money, it's often the case that the owners simply don't know how important their buildings are."