Q:

What is the algorithm that R uses in data frame filtering?

For example, I have a data frame, and I want to subset it according to specific conditions:
df[df$gender == "woman" & df$age > 40, ]

What is the algorithm behind this filtering in R?

A:

During the execution of the code df[df$gender == "woman" & df$age > 40, ] the following happens:

df$gender extracted.
df$gender == "woman" evaluated, boolean vector returned.
df$age extracted.
df$age > 40 evaluated, boolean vector returned.
Logical by-element AND applied to each element of vectors in Step 2 and Step 4.
Rows from df which have TRUE flag in Step 5 are extracted.

In all above cases backend C\C++ functions are called. E.g. [] subset function calls for do_subset in subset.c.
You can study mapping between R functions and it's C back-end in names.c
For further details you can consult Accessing R Source

