Q:

calling methods for string prototype

can someone explain why executing this below result in getting toUpperCase function?
("string")["toUpperCase"]

function toUpperCase()

A:

EDIT: I found the question interesting, so I will try to provide a detailed answer.
Also, allow me to change your example to:
("hello")["toUpperCase"]

as the use of "string" could mislead programming newcomers (e.g. "string" is a string).

SHORT ANSWER:
Because ("hello") gets temporary converted to a String object whose property ["toUpperCase"] has a reference to a Function instance. When you do:
console.log( ("hello")["toUpperCase"] );

you are accessing the value of that property, i.e., a reference to the toUpperCase function.

LONG ANSWER:
Let's break ("hello")["toUpperCase"] into pieces:

("hello") evaluates as a string literal (a primitive data type).
["toUpperCase"] access a method (a property containing a function definition).

("hello")
We keep on breaking down:
"hello" is, undoubtedly, a string literal:
console.log( typeof "hello" ); // Logs "string"

When we wrap it in parentheses (grouping operator) it becomes a parenthesized expression. As you can read in the ECMAScript specification:

ParenthesizedExpression : ( Expression )
Return the result of evaluating Expression. This may be of type Reference.

It means that a parenthesized expression evaluates to the value of the expression within the parentheses. As result of it:
console.log( "hello" === ("hello") ); // Logs "true"
console.log( typeof ("hello") ); // Logs "string"

So we can conclude that ("hello") evaluates as a string literal.

["toUpperCase"]
As you can read in MDN, the bracket notation is as valid as the dot notation when accessing object's properties:

Property accessors provide access to an object's properties by using
  the dot notation or the bracket notation.
Syntax
object.property
object["property"]

Now, maybe you are wondering why I talk about properties if toUpperCase is a method. If you keep on reading the same link above:

One can think of an object as an associative array (a.k.a. map,
  dictionary, hash, lookup table). The keys in this array are the names
  of the object's properties. It's typical when speaking of an object's
  properties to make a distinction between properties and methods.
  However, the property/method distinction is little more than a
  convention. A method is simply a property that can be called, for
  example if it has a reference to a Function instance as its value.

Actually, the ECMAScript specification defines method as "function that is the value of a property".
So we can conclude that ["toUpperCase"] is a property accessor (provide access to an object's property).

("hello")["toUpperCase"] together again
We have seen that ("hello")["toUpperCase"] is equivalent to "hello".toUpperCase, actually:
console.log( ("hello")["toUpperCase"] === "hello".toUpperCase ); // Logs "true"

But, why is this working? "hello" is not an object! it is a string literal (a primitive)!
That is completely right:
console.log( typeof "hello" ); // Logs "string"
console.log( typeof new String("hello") );  // Logs "object"

but JavaScript will coerce that literal into an object for you, so you can use the toUpperCase method. As you can read in the definition for the String global object in MDN:

String literals (denoted by double or single quotes) and strings
  returned from String calls in a non-constructor context (i.e., without
  using the new keyword) are primitive strings. JavaScript automatically
  converts primitives to String objects, so that it's possible to use
  String object methods for primitive strings. In contexts where a
  method is to be invoked on a primitive string or a property lookup
  occurs, JavaScript will automatically wrap the string primitive and
  call the method or perform the property lookup.

That object will be discarded afterwards (MDN - Grammar and types):

You can call any of the methods of the String object on a string
  literal value—JavaScript automatically converts the string literal to
  a temporary String object, calls the method, then discards the
  temporary String object.

So we can conclude that ("hello")["toUpperCase"] is accessing the toUpperCase method/property from the (temporary) String object "hello".

So, why am I getting a function?
Because that is what is inside the object's property: it is the value of the property. It is said that functions in JavaScript are "first class citizens", meaning that you can pass them around like any other piece of data. You can do things like this:
var foo = new String( "I am foo" );
foo.upperCase = ("hello")["toUpperCase"];   
console.log( foo.upperCase() ); // Logs "I AM FOO"

If what you want is to invoke the method, you need to use the function call operator ().
console.log( ("hello")["toUpperCase"]() ); // Logs "HELLO"

