Q:

Pointer memory usage

I may be wrong on these facts so correct me!
A pointer points to a memory address, if you went to this memory address you would find a byte ? A pointer also points to the lowest address of the memory segment that it points too, so if it points to an 8 byte memory segment, that starts at 0x0 and ends at 0x7, it will point to 0x0 ?
How does a pointer know the size of memory it points at? So if a pointer points to a segment of memory that is 128 bytes in size and the pointer is cast to a different type, what happens to the size of the memory segment?

A:

How does a pointer know the size of memory it points at ? 

It doesn't.

So if a pointer points to a segment of memory that is 128 bytes in size and the pointer is cast to a different type, what happens to the size of the memory segment ?

The memory's still there, but since you cast away the only information you had about the object residing there, well, y'know. That's it. You don't know.
That's why people use sizeof when aliasing objects through a char* to get at the underlying bytes:
std::ofstream os("some-binary-file");
const double d = 0.1234;

os.write((char*)&d, sizeof(double));
//                  ^^^^^^^^^^^^^^
// 
// Otherwise os.write has no way of knowing how much to write

Of course, if you were to chuck around a double*, every piece of code using that pointer will assume that it points to one or more "blocks" of memory precisely sizeof(double) bytes wide. If it doesn't, then that's your fault.
tl;dr: Telling the program how many pointee bytes it has to work with is your responsibility.

