now the only time the two boys saw each other for more than a few moments was in the late afternoon when pip would stay at the top and watch , with more than a little envy , as the other boys went swimming in the cove below .