Castles in the Air

I used to always think this song was about a man leaving his wife until recently I realized that he never refers to the woman as such. Probably she was just some girlfriend which makes it somehow less melancholy.
On the other hand maybe they are married. Cause who's the other person that's supposed to be listening to and then relating the message to the girl - the man's attorney?

song: Castles in the Air • artist: Don McClean