Conservative MP Alex Burghart has defended not visiting Northern Ireland during his time as shadow secretary.