Let's say I'm programming in Java or Python or C++ for a simple problem, could be to build an TCP/UDP echo server or computation of factorial. Do I've to bother about the architecture details, i.e., if it is 32 or 64-bit?
IMHO, unless I'm programming something to do with fairly low-level stuff then I don't have to bother if its 32 or 64 bit. Where am I going wrong? Or am I correct???
As long as you do things correctly, you almost never need to know for most languages.  On many, you never need to know, as the language behavior doesn't vary (Java, for example, specifies the runtime behavior precisely).
In C++ and C, doing things correctly includes not making assumptions about int.  Don't put pointers in int, and when you're doing anything with memory sizes or addresses use size_t and ptrdiff_t.  Don't count on the size of data types:  int must be at least 16 bits, almost always is 32, and may be 64 on some architectures.  Don't assume that floating-point arithmetic will be done in exactly the same way on different machines (the IEEE standards have some leeway in them).
Pretty much all OSes that support networking will give you some way to deal with possible endianness problems.  Use them.  Use language facilities like isalpha() to classify characters, rather than arithmetic operations on characters (which might be something weird like EBCDIC).  (Of course, it's now more usual to use wchar_t as character type, and use Unicode internally.)
Would one have to know the machine architecture to write code?