Q:

What is the physical interpretation of Lowpass/Highpass filtering?

In Image/Signal processing context we have seen/we know that if there are say 4 samples and if we take an average of those 4 samples, then we say that the result sample is a low pass filtered output sample. This is very relevant in Image processing context. Now then, what is the operation which implies a high pass filtering, is it finding some max of the 4 samples and putting that as the output sample, or what other mathematical operations implies high pass filtering as averaging implies low pass filtering.

A:

The example you gave of taking 4 samples and taking the average of it is sort of a poor-man's low-pass filter. Generally things aren't as simple as that. But for understanding sake there is some value in using these simple examples.
A low pass filter is indeed like taking 4 samples and taking an average of it. Ex:
samples = [6 1 -10 -4];
avg_value = mean(samples) = -1.75

The high pass filter is removing the "DC" term. Or more generally, it is removing the data that isn't changing. A simple way of thinking of this is to subtract your avg_value from every sample. Ex:
samples = [6 1 -10 -4];
avg_value = mean(samples) = -1.75;
high_pass = samples-avg_value;
high_pass: [7.75 2.75 -8.25 -2.25]

Now if you take the average of the "high passed" signal you end up getting 0.

These two 'filters' that I mention are both to the extreme in that you have one filter that only gives you DC and another filter that only removes DC. Basically what you end up getting is this ideal filter where the low-pass filter gives you the green and the high pass filter gives you the yellow.

Most of the filters you will use will have a response that looks more like this for a low pass:

and this for a high pass:

A:

Firstly, an average is a very specific low-pass filter.
High-pass filtering means keeping fast-changes and discarding the "gradual changes".  Differentiation is one classic mathematical way of doing this.
In the discrete domain, if you convolve a signal vector with $(\begin{matrix} 1 & -1 \end{matrix})$ you see peaks wherever the signal changes quickly.  This is a high-pass filter.
High-pass filtering is also called "edge-detection" in image processing circles.

A:

In image processing, low pass filter makes images smoother, and more blurry since it averages the neighborhood of the pixel. 
High pass filter makes edges become more visible and sharper since it detects edges in images. This is because where edges occur is the most dramatic change occurs in the images. Low pass tries to decrease this dramatic increase or decrease in image by averaging the the neighborhood whereas high pass filter makes it more visible by subtracting the pixel values.  

