Let's say I'm programming in Java or Python or C++ for a simple problem, could be to build an TCP/UDP echo server or computation of factorial. Do I've to bother about the architecture details, i.e., if it is 32 or 64-bit?
IMHO, unless I'm programming something to do with fairly low-level stuff then I don't have to bother if its 32 or 64 bit. Where am I going wrong? Or am I correct???
You will need to care about "endian-ness" only if you send and receive raw C structs 
over the wire like

ret = send(socket, &myStruct, sizeof(myStruct));

However this is not a recommended practice.
It's recommended that you define a protocol between the parties such it doesn't matter
the parties' machine architectures.
Would one have to know the machine architecture to write code?