Hi,

I have a *list of genes* (`long_list`) from which I want to filter a list with *genes of interest* (`goi`) to retrieve a *filtered list of genes* (`genes_filtered`). The `goi`s are all expected to be contained in the `long_list`.

However, when I filter for the `goi`s, I always retrieve four extra genes. 

    genes_filtered <- long_list[long_list$ensembl %in% goi$ensembl,]

Dimensions of the lists:

    dim(long_list)[1]
    > 15251
    dim(goi)[1]
    > 11221
    dim(genes_filtered)[1]
    > 11225 #Should be 11221 (same as goi)


I have tried the following to get to the bottom if this.

Checking **duplicates** in `genes_filtered`, `long_list`, and `goi`:

    dim(genes_filtered[duplicated(genes_filtered$ensembl),])[1]
    > [1] 0
    dim(long_list[duplicated(long_list$ensembl),])[1]
    > [1] 0
    dim(goi[duplicated(goi$ensembl),])[1]
    > NULL

Checking **missing values**:

    sum(is.na(genes_filtered))
    > [1] 0
    sum(is.na(long_list))
    > [1] 0
    sum(is.na(goi))
    > [1] 0

Checking **values contained in `genes_filtered` but not in `goi`**:
    
    # 1: Using lists
    genes_filtered[!(genes_filtered$ensembl %in% goi$ensembl)]
    data frame with 0 columns and 11125 rows
    
    # 2: Extracting columns first from lists
    f <- genes_filtered$ensembl
    g <- goi$ensembl

    g[!(g %in% f)]
    [1] "ENSG00000283208" "ENSG00000284292" "ENSG00000262633" ...

Method 2 retrieves a list of in total 96 genes, which is not expected.

Can anyone explain why the filtering method at the top of the post does not work and possibly suggest a correct way?


I can't believe you and I missed this. It took me a while to workout what the issue was:

    length(ll) # 15202
    length(goi) # 11221
    sum(ll %in% goi) # 11125

In case you missed it:

    11221
    11125

The difference? 96. You said:

> Method 2 retrieves a list of in total 96 genes, which is not expected.

Not sure why we discounted this, but method 2 is correct. There's a difference of 96 genes between `goi` and `long_list`. Here's another method that shows this:

    library(dplyr)
    goi_df = data.frame(id = goi,
                        goi_id = 1:length(goi))
    ll_df = data.frame(id = ll,
                        ll_id = 1:length(ll))
    combine = left_join(ll_df, goi_df, by = "id")
    
    sum(is.na(combine$goi_id)) # 4077
    
    15202 - 4077 # 11125

So which genes of interest are missing?

    setdiff(goi, combined_subset$id)

>  [1] "ENSG00000283208" "ENSG00000284292" "ENSG00000262633"
> "ENSG00000258472" "ENSG00000276017" "ENSG00000283809"
> "ENSG00000277971" "ENSG00000277639"  [9] "ENSG00000269755"
> "ENSG00000268400" "ENSG00000258311" "ENSG00000261341"
> "ENSG00000268750" "ENSG00000279765" "ENSG00000226690"
> "ENSG00000284526" [17] "ENSG00000267740" "ENSG00000261884"
> "ENSG00000267426" "ENSG00000170846" "ENSG00000283580"
> "ENSG00000267314" "ENSG00000283782" "ENSG00000283761" [25]
> "ENSG00000214265" "ENSG00000183889" "ENSG00000262165"
> "ENSG00000275063" "ENSG00000277856" "ENSG00000182584"
> "ENSG00000267120" "ENSG00000259529" [33] "ENSG00000258465"
> "ENSG00000203546" "ENSG00000282246" "ENSG00000187186"
> "ENSG00000278384" "ENSG00000124593" "ENSG00000273748"
> "ENSG00000277263" [41] "ENSG00000285085" "ENSG00000256591"
> "ENSG00000173915" "ENSG00000156411" "ENSG00000214654"
> "ENSG00000284976" "ENSG00000151131" "ENSG00000130921" [49]
> "ENSG00000174206" "ENSG00000125149" "ENSG00000088854"
> "ENSG00000285217" "ENSG00000171159" "ENSG00000104957"
> "ENSG00000198003" "ENSG00000160124" [57] "ENSG00000018610"
> "ENSG00000187866" "ENSG00000156504" "ENSG00000156500"
> "ENSG00000051009" "ENSG00000151553" "ENSG00000158863"
> "ENSG00000285382" [65] "ENSG00000275464" "ENSG00000287542"
> "ENSG00000232593" "ENSG00000162929" "ENSG00000276033"
> "ENSG00000274847" "ENSG00000072415" "ENSG00000105926" [73]
> "ENSG00000198899" "ENSG00000228253" "ENSG00000198804"
> "ENSG00000198712" "ENSG00000198938" "ENSG00000198727"
> "ENSG00000198888" "ENSG00000198763" [81] "ENSG00000198840"
> "ENSG00000198886" "ENSG00000212907" "ENSG00000198786"
> "ENSG00000198695" "ENSG00000258724" "ENSG00000228049"
> "ENSG00000285437" [89] "ENSG00000143303" "ENSG00000205045"
> "ENSG00000102125" "ENSG00000285053" "ENSG00000283268"
> "ENSG00000011638" "ENSG00000243667" "ENSG00000225528"

Tricky tricky. Hope this helps.