Bob,
When I was a kid,  I was a klutz when it came to the "ball sports."  I was the last pick when it came to baseball, football or basketball.  But I excelled at swimming and competed from junior high through college.

And I still swim.  I credit a regular workout in the pool with helping me endure the long work days and challenges I faced as a TV News Photographer at KEZI and KVAL.  If the corporate owner of two of our three local network affiliates hadn't "downsized" my position, I feel that I have easy done my job well into my 70s with the energizing benefits I have reaped from routine swimming.

Now that you've worked up to a mile, you should seek out someone who could provide you with a few pointers about technique.  Improving even a slight alteration in how you position your head (when turning to breathe), arms, hands and legs as you move through the water can really enhance your endurance and enjoyment while swimming laps.  I have a contact for someone who could help.