Political leaders on the Isle of Wight have hit back after a community group accused them of letting Sandown “rot.”