Just a reminder that the Harbor structures need possession flag posts on them, just like towns, cities and forts have. Otherwise, there is no possession flag post in the region.
Agree. It is especially annoying in the West when thanks to cavalry those harbors change hands several times. Now you have to either read the log closely each turn or hover your tooltip to learn who owns the place.