text:
Single cell tutorial
====================

**Important update**: We now recommend the use of `alevin` for droplet-based
scRNA-Seq (e.g 10X, inDrop etc). `alevin` extends the directional
method used in `UMI-tools` to correct UMI errors with droplet
scRNA-Seq within a framework that also enables quantification using
multi-mapped reads. `alevin` is an accurate, fast and convenient
end-to-end tool to go from fastq -> count matrix. See
[alevin documentation](https://salmon.readthedocs.io/en/latest/alevin.html)
and [alevin pre-print](https://www.biorxiv.org/content/10.1101/335000v2) for more information

This tutorial will go through an end to end analysis for single cell analysis using UMI-tools. Before you start, you will need:
*  An installed copy of UMI-tools (see the installation guide)
*  [The STAR aligner](https://github.com/alexdobin/STAR)
*  A STAR index for the human genome
*  A transcriptome annotation.
*  The [Subread package](http://subread.sourceforge.net/), version 1.5.3 or greater.

This tutorial will use an example dataset generated by 10X Genomics on their Chromium platform. At the end of the tutorial we cover variations on this workflow. We will continue to expand this section and cover other other data types here as well.

With 4 cores at your disposal, the whole thing should be do-able in less than an hour for this dataset.

The below assumes you are working in a linux environment. Some changes might be necessary if you are working in OSX, notably `wget` should be replace with `curl -O`

Outline of the process:

| Step | Inputs | Output |
| -------- | -------- | -------- |
| Find Cell barcode whitelist    | Read 1 fastq     | whitelist.txt     |
| Extract CB/UMIs and filter CBs | R1 + R2 fastqs + whitelist.txt | extracted fastqs |
| Map reads | extracted fastqs  | BAM |
| Assign reads to genes | BAM + transcriptome GTF | BAM
| Count unique reads per genes per cell | BAM | Counts.txt

TL;DR
-----
```bash
#! /bin/env bash
# Step 1: get data
wget http://cf.10xgenomics.com/samples/cell-exp/1.3.0/hgmm_100/hgmm_100_fastqs.tar;
tar -xf hgmm_100_fastqs.tar;
cat fastqs/hgmm_100_S1_L00?_R1_001.fastq.gz > hgmm_100_R1.fastq.gz;
cat fastqs/hgmm_100_S1_L00?_R2_001.fastq.gz > hgmm_100_R2.fastq.gz;

# Step 2: Identify correct cell barcodes
umi_tools whitelist --stdin hgmm_100_R1.fastq.gz \
                    --bc-pattern=CCCCCCCCCCCCCCCCNNNNNNNNNN \
                    --set-cell-number=100 \
                    --log2stderr > whitelist.txt;

# Step 3: Extract barcdoes and UMIs and add to read names
umi_tools extract --bc-pattern=CCCCCCCCCCCCCCCCNNNNNNNNNN \
                  --stdin hgmm_100_R1.fastq.gz \
                  --stdout hgmm_100_R1_extracted.fastq.gz \
                  --read2-in hgmm_100_R2.fastq.gz \
                  --read2-out=hgmm_100_R2_extracted.fastq.gz \
                  --whitelist=whitelist.txt;
# Step 4: Map reads
STAR --runThreadN 4 \
     --genomeDir hg38_noalt_junc85_99.dir \
     --readFilesIn hgmm_100_R2_extracted.fastq.gz \
     --readFilesCommand zcat \
     --outFilterMultimapNmax 1 \
     --outSAMtype BAM SortedByCoordinate;

# Step 5: Assign reads to genes
featureCounts -a geneset.gtf \
              -o gene_assigned \
              -R BAM Aligned.sortedByCoord.out.bam \
              -T 4;
samtools sort Aligned.sortedByCoord.out.bam.featureCounts.bam -o assigned_sorted.bam;
samtools index assigned_sorted.bam;

# Step 6: Count UMIs per gene per cell
umi_tools count --per-gene --gene-tag=XT --assigned-status-tag=XS --per-cell -I assigned_sorted.bam -S counts.tsv.gz

```

Step 1: Obtaining the data
--------------------------
The dataset was produced from around 100 cells from a mixture of human and mouse cells. The download is 760MB, so make sure you've got space. It can be downloaded thus:

```
$ wget http://cf.10xgenomics.com/samples/cell-exp/1.3.0/hgmm_100/hgmm_100_fastqs.tar
$ tar -x hgmm_100_fastqs.tar
```

out of the archive you will get a directory called `fastqs` that will contain data from 8 sequencing lanes, with three files for each lane: a read 1 file, a read 2 file and file containing the Sample Barcodes. We are not interested in the sample barcodes files, but for the read 1 and read2 files, we need to combine the eight lanes into one. By the magic of gzip, we can do with `cat`:

    $ cat fastqs/hgmm_100_S1_L00?_R1_001.fastq.gz > hgmm_100_R1.fastq.gz
    $ cat fastqs/hgmm_100_S1_L00?_R2_001.fastq.gz > hgmm_100_R2.fastq.gz

If we look at the content of of the files we will see that the first read in the pair contains 26nt, which correspond to a 16nt Cell barcode (CB) and 10nt the Unique Molecular Identifier (UMI):

    $ zcat hgmm_100_R1.fastq.gz | head -n2
    @ST-K00126:308:HFLYFBBXX:1:1101:25834:1173 1:N:0:NACCACCA
    NGGGTCAGTCTAGTGTGGCGATTCAC
    +
    #AAFFJJJJJJJJJJJJJJJJJJJJJ

    -------CB-------|---UMI---

We will need to remember this for the next two steps

Step 2: Identifying the real cells
---------------------------------

Cell barcodes are short nucleotide sequences, very much like UMIs, except instead of identifying independent molecules, they identify independent cells. We generally observe more of them in an experiment than there were cells. This could be for several reasons, including sequencing or PCR errors and the sequencing of empty droplets or those containing contaminants. Thus we must identify which cell barcodes we wish to use downstream. UMI-Tools `whitelist` command is used to produce a list of CB to use downstream.

`whitelist` currently allows the common method of taking the top X most abundant barcodes. X can be estimated automatically from the data using the `knee` method (for more detail see this [blog post](https://cgatoxford.wordpress.com/2017/05/18/estimating-the-number-of-true-cell-barcodes-in-single-cell-rna-seq/)). However, it is just an estimate and for this data we've been told that there were 100 cells, so we can just supply that number (see variations section for performing the estimation for data sets where cell number is unknown).

     umi_tools whitelist --stdin hgmm_100_R1.fastq.gz \
                         --bc-pattern=CCCCCCCCCCCCCCCCNNNNNNNNNN \
                         --set-cell-number=100 \
                         --log2stderr > whitelist.txt

A couple of things about this command:

### Input files
Firstly, note that FASTQ file that contains the barcodes is passed to  `--stdin`, this tells `whitelist` to read the data from this file rather than from stdin. This of course means you could pass data using standard pipes.

### Plots
The `--plot-prefix` option tells `whitelist` to output summary plots for the frequency of each CB. We always recommend running with this option in order to visualise whether the number of cells accepted seems reasonable. See [Variations](Automatic estimation of cell number) for a more complete explanation of what these plots show.

### Specifying barcode locations
Second, the `--bc-pattern`. This tells `whitelist` where to find the CB and UMI in the read sequence. By default we assume the barcodes are at the 5' end of the read (this can be changed with `--3prime`). We then use `C` characters to show where CB bases are and `N` characters to show were UMI bases are. Thus, in the above we have 16 `C`s followed by 10 `N`s to denote that the first 16 bases of the read are CB bases and the second 16 are UMI bases. Alternatively, you can also define the barcode pattern using a regex instead (--extract-method=regex) in which there are named groups to define the positions for the cell barcode and UMI. For example we would change the above command to:

    umi_tools whitelist --stdin hgmm_100_R1.fastq.gz \
                        --bc-pattern='(?P<cell_1>.{16})(?P<umi_1>.{10})' \
                        --extract-method=regex \
                        --set-cell-number=100 \
                        --log2stderr > whitelist.txt

This interface is very powerful and flexible and allows for the specification of all sorts of interesting things, like variable length CBs (UMIs do have to be a fixed length) and tolerant linker sequences (see the inDrop example in [Variations](#Barcode-extraction-for-inDrop) and `umi_tools whitelist --help`).

### Specifying outputs

By default UMI-tools outputs everything, final output, logging info and progress report, to the standard out pipe. Downstream sections will take this output just fine, but you might want to put the log somewhere else, like a separate file, or on the terminal. This can be achieved in one of several ways:

* You could redirect the final output using the `--stdout` or `-S` options. The final output will be directed to a file and the log will continue to come on stdout.
* You could redirect the log to a file with `--stdlog` or `-L`. The log and progress will be saved to a file and the final output sent to the standard out.
* You can redirect the log and progress to the standard err using `--log2stderr` as above.
* You can switch off the log/progress with `-v 0`

This applies to all the UMI-Tools commands.

### Using UMI counts rather than read counts in umi_tools whitelist
Many published protocols rank CBs by the number of reads the CBs appear in. However you could also use the number of unique UMIs a CB is associated with. Note that this is still an approximation to the number of transcripts captured because the same UMI could be associated with two different transcripts and be counted as independent. Activate this with `--method=umis`.

### Contents of `whitelist.txt`

The output of the whitelist command is a table containing the accepted CBs. It has four columns:
1. The accepted CB
2. Comma separated list of other CBs within an edit distance of the CB in columns 1 and >1 edit away from any other accepted CB.
3. The abundance (read or UMI count) of the accepted.
4. Comma separated list of abundances for the CBs in column 2

e.g:

```
$ head whitelist.txt
AAAGATGAGAAACGAG	AAAAATGAGAAACGAG,AAACATGAGAAACGAG,...	53122	4,6,...
AAAGCAAGTACCTACA	AAAACAAGTACCTACA,AAACCAAGTACCTACA,...	36255	2,3,...
AACACGTCAGCGTAAG	AAAACGTCAGCGTAAG,AACAAGTCAGCGTAAG,...	53133	4,11,...
```
(I have truncated columns 2 and 4 for easy viewing)

As we asked for 100 cells, this file should contain 100 lines:

```
$ wc -l whitelist.txt
100 whitelist.txt
```

Step 3: Extract the barcodes and filter the reads
-------------------------------------------------

The next step is to extract the CB and UMI from Read 1 and add it to the Read 2 read name. We will also filter out reads that do not match one of the accepted cell barcode.

The most basic form of this is executed with:

```
umi_tools extract --bc-pattern=CCCCCCCCCCCCCCCCNNNNNNNNNN \
                  --stdin hgmm_100_R1.fastq.gz \
                  --stdout hgmm_100_R1_extracted.fastq.gz \
                  --read2-in hgmm_100_R2.fastq.gz \
                  --read2-out=hgmm_100_R2_extracted.fastq.gz \
                  --whitelist=whitelist.txt
```

The `--bc-pattern` and `--stdin` options are as before. Note that we send the standard out (which contains the extract Read 1s) to the a file (add `.gz` to the end of the name will trigger automatic compression). `--read2-in` and `--read2-out` specify the output files for read 2. Finally `--whitelist` passes the list of accepted CBs generated in the previous step and tells `extract` to only output those reads that contain accepted CBs.

As the end of the log makes clear, the command above took about 20 minutes

```
2017-08-10 16:29:45,449 INFO Parsed 7100000 reads
2017-08-10 16:30:01,896 INFO Input Reads: 7197662
2017-08-10 16:30:01,896 INFO Reads output: 5963891
2017-08-10 16:30:01,896 INFO Filtered cell barcode: 1233771
# job finished in 1175 seconds at Thu Aug 10 16:30:01 2017 -- 1153.42 24.13  0.00  0.00 -- f5227f66-4f36-4862-9c7a-87caf3f81870
```

There should now be two output files in your directory: `hgmm_100_R1_extracted.fastq.gz` and `hgmm_100_R2_extracted.fastq.gz`. Looking at the first read we see:

```
$ zcat hgmm_100_R1_extracted.fastq.gz | head -n4
@ST-K00126:308:HFLYFBBXX:1:1101:31345:1261_AACTCTTGTTCTGAAC_CGGTTGGGAT 1:N:0:NACCACCA

+
```

The reads are now empty because the CBs and UMIs have now been moved from the read sequence to the read name, and there is nothing left. We will not need this file again.

If we look at the Read 2:

```
$ zcat hgmm_100_R2_extracted.fastq.gz | head -n4
@ST-K00126:308:HFLYFBBXX:1:1101:31345:1261_AACTCTTGTTCTGAAC_CGGTTGGGAT 2:N:0:NACCACCA
CCTTTTTGGAACCAACAATAGCAGCTCCATTTCTGGAGTCTGGGTCTTCCGAGGCCAGGAGCTCGCCTTTCCGCCGAGCCCAGATTGGCAGGTGGACT
+
A<<AFAFFFJJJJFJJJFA<7<FJF-AAJF7-FFF<FA7AFFFJ-77<JJFFFJJJJFAJFJ7-7AJ-7-FJJJ--)7-77F-F--AAAJAA-7-7F7
```

We can see that the CB and UMI from read 1 has been added to the name of read 2 as well.

These reads are now ready to map!

### Discarding read 1
There are many single cell RNA-Seq techniques where read 1 only contains barcodes, as above. If we only want to write out the extracted read 2 file we can supply the `--read2-stdout` option to output read 2 on the stdout and discard read 1.

Step 4: Mapping reads
---------------------

At this point there are two different ways we might proceed. We could map to the transcriptome, where each each contig represents a different transcript. Alternatively we could map to the genome and then assign reads to genes. While transcriptome mapping allows for easier downstream analysis, we recommend mapping to the genome because it doesn't force reads to map to transcripts when a better match outside an annotated gene exists. However, there are cases when transcriptome mapping should be favoured, and we discuss this in the [Variations](#Mapping-to-the-transcriptome-rather-than-genome) section.

We are going to use STAR here as our mapper, but you can substitute whichever splice-aware aligner you prefer. We are mapping to an index built on the human NCBI38 reference, with the alternate contigs removed (referred to by NCBI as the "analysis read set") using junctions from ENSEMBL85 and a 99nt overhang. We do not allow multimapping reads.

```bash
$ STAR --runThreadN 4 \
       --genomeDir hg38_noalt_junc85_99.dir \
       --readFilesIn hgmm_100_R2_extracted.fastq.gz \
       --readFilesCommand zcat \
       --outFilterMultimapNmax 1 \
       --outSAMtype BAM SortedByCoordinate
```

If we have a look at the stats for this, we will see that 63.02% of reads mapped (depending on exactly how you've built your STAR index you may get a slightly different number). For 8.1% of these this is because of mapping to more than one location. The rest is probably because the sample contains both mouse and human cells, but we are only aligning to the human genome. In fact, given that, the mapping rate looks a bit on the high side: we are probably mapping some reads to the human genome that actually came from the mouse genome. Ideally we would perform this mapping using references that combined both the mouse and human genomes/transcriptomes.

On our systems, using 4 threads, this mapping takes about 7 minutes.

Step 5: Assigning reads to genes
--------------------------

Many of the single cell techniques that use UMIs are 3' tag count methods. This is important because it means that the UMIs are generally added before PCR and then fragmentation. This means that two reads from the same gene might have different mapping locations and still represent duplicates, as long as they come from the same gene. Therefore, UMI-tools need to know which gene each read came from. When reads are aligned to the transcriptome, this is easy as it is given by the contig (and possibly a gene to transcript look up table). However, with a genome alignment we need to assign reads to genes.

We can do this with the `featureCounts` tool from the `subread` package. As well as outputting a table of (undeduplicated) counts, we can also instruct `featureCounts` to output a BAM with a new tag containing the identity of any gene the read maps to.

**warning**: This only works on featureCounts from subread 1.5.3 and above, which was released July 2017. If you already have subread it could well need updating

You will need a geneset to assign genes to. Here ours is called `geneset.gtf`. It is the same geneset used to generate the mapping index above, and is based on the GENCODE annotations.

```
$ featureCounts -a geneset.gtf -o gene_assigned -R BAM Aligned.sortedByCoord.out.bam -T 4

```

Unfortunately the BAM output by featureCounts is no longer sorted, so we now need to sort the BAM.

```
$ samtools sort Aligned.sortedByCoord.out.bam.featureCounts.bam -o assigned_sorted.bam
$ samtools index assigned_sorted.bam
```

We now have a sorted, indexed set of alignments assigned to the appropriate genes and we can now count the number of distinct UMIs mapping to each gene in each cell.

Step 6: Counting molecules
-------------------------

We are finally ready to process the UMIs aligned to each gene in each cell to find the number of distinct, error corrected UMIs mapping to each gene.

```
$ umi_tools count --per-gene --gene-tag=XT --assigned-status-tag=XS --per-cell -I assigned_sorted.bam -S counts.tsv.gz
```

Since we want to count the number of UMIs per gene, and the gene assignment is encoded in the XT tag, we use the `--per-gene --gene-tag=XT` options. FeatureCounts also uses the XS tag to describe the gene assignment so we use the `--assigned-status-tag=XS` option. Any reads with an assignment matching `^[__|Unassigned]` (default value for `--skip-tags-regex`) will be skipped. `--per-cell` tells UMI-tools to also consider the CB and produce a separate count for each cell. The counts are output in a table with three columns: the gene_id, the cell barcode and the count of deduplicated UMIs.

Running this command on this input takes 72 seconds on our system.

```$ zcat counts.tsv.gz | head
$ zcat counts.tsv.gz | head
gene	cell	count
ENSG00000000003	AAAGATGAGAAACGAG	3
ENSG00000000003	AACTCTTGTTCTGAAC	4
ENSG00000000003	ACACCGGGTACGACCC	2
ENSG00000000003	ACACTGAGTCGGGTCT	5
ENSG00000000003	ACTATCTCAAGGTGTG	2
ENSG00000000003	ACTGTCCCATATGGTC	11
ENSG00000000003	ACTGTCCTCATGCTCC	5
ENSG00000000003	AGAGCTTCACGACGAA	2
ENSG00000000003	AGCGGTCTCGATAGAA	2
```

If you'd rather have this as a matrix, which each cell as a sepearte column, you can specify `--wide-format-cell-counts`. You can also output deduplicated reads using the `dedup` command in place of `count`. You can even output all reads, with each assigned to an UMI group, using the `group` command, although see below for some important points.

Time and memory for larger data sets
-------------------------------------

While the above was pretty quick, 7 million reads over 100 cells is a pretty tiny dataset. The most memory intensive part was the `STAR` alignment, and this is unlikely to change with larger datasets. Below we provide some figures for larger datasets. These times will obviously vary depending on your setup, in particular many of these tasks are I/O bound, and the nature of your storage will make a big difference. Running on local storage, particularly if its SSD, will probably give substantially quicker performance. Conversely if storage is accessed over standard network connections, it will probably be much slower.

| Step | 10X 100 cells | 10X 1,000 cells |
| -------- | -------- | -------- |
| Whitelist     | 00:02* / 125Mb     | 00:21 / 261 Mb    |
| Extract | 00:10 96Mb  | 03:33* / 100 Mb   |
| Mapping | 00:05 / 17Gb | 00:23 / 17Gb |
| Assigning reads | 00:01 / 203 Mb | 00:04 / 137 Mb |
| Sorting/Indexing | 00:01 / 809Mb** | 00:12 / 830Mb** |
| Counting | 00:02 / 203 Mb | 00:09 / 1.1Gb*** |

  \* Much of this time is spent compressing the output. Outputting to uncompressed files is significantly faster.

  \** memory used by `samtools sort` can be specified with the `-m` option.

  \*** memory used counting is significaitonly reduced by mapping to the transcriptome rather than the genome.

Variations
----------

### Barcode extraction for Drop-seq
For [Drop-seq](2), the current protocol includes a 12bp CB, followed by an 8bp UMI. We can therefore extract the barcodes using the following options:

```
--extract-method=string
--bc-pattern=CCCCCCCCCCCCNNNNNNNN
```

### Barcode extraction for inDrop
The CBs for [inDrop](1) are different lengths. Thus, we must use `--extract-method=regex` since we can no longer encode the barcode pattern in a simple string. If we look at the sequence of an inDrop read 1, we can demark the barcodes as follows:

```
TGAACATCTATGAGTGATTGCTTGTGACGCCTTGAGCCCATCCTGCTTTTTTTTT
---CB-1---|------ADAPTER--------|--CB-2-|-UMI-|poly(T)
```
The CB is split into two halves. CB-1 can be 8-12bp, the adapter sequence is always "GAGTGATTGCTTGTGACGCCTT", CB-2 is 8bp, UMI is 6bp and the read ends in a poly(T) seqeunce. In the case above, the barcode sequences are:

CB-1: TGAACATCTAT
CB-2: GAGCCCAT
CB (CB-1 + CB-2): TGAACATCTATGAGCCCAT
UMI: CCTGCT

To encode this in the barcode pattern, we must provide the option `--extract-method=regex` and provide `--bc-pattern` with a regex with groups named accordingly. Each group must have a integer suffix:
`--bc-pattern="(?P<cell_1>.{8,12})(?P<discard_1>GAGTGATTGCTTGTGACGCCTT)(?P<cell_2>.{8})(?P<umi_1>.{6})T{3}.*"`

This translates to: 8-12 characters (group="cell_1"), followed by "GAGTGATTGCTTGTGACGCCTT" (group="discard_1"), followed by 8 characters (group="cell_2"), followed by 6 characters (group="umi_1"), followed by 3 or more "T"s.

The other benefit of using a regex is that we can perform ['fuzzy'](https://pypi.python.org/pypi/regex/). For example, the inDrop adapter sequence is 22bp long so it may sometimes contain base calling errors. We can allow usp to two substituions like so:

`--bc-pattern="(?P<cell_1>.{8,12})(?P<discard_1>GAGTGATTGCTTGTGACGCCTT){s<=2}(?P<cell_2>.{8})(?P<umi_1>.{6})T{3}.*"`

It should be possible to encode any conceivable pattern of barcodes using a regex.

### Automatic estimation of cell number
In [Step 2](#Step-2:-Identifying-the-real-cells) above, we generated a whitelist of accepted ("true") CBs and provided the option `--set-cell-number=100` since we a prior belief about the number of cells we had sequenced. However, what if we didn't know how many cells we had actually sequenced? Well, in that case, we can leave out the `--set-cell-number` option and let `whitelist` estimate the number of true CBs automatically. Even better, if we have an estimate on the number of cells we have input into the system and we know the capture rate is at least 10% (this assumption holds true for inDrop and 10X Chromium platforms), we can provide the estimated number of input cells as an upper limit on the number of true CBs using the `--expect-cells` option. So, imagine with our experiment above, we estimated that around 200 cells were input into the system, we can generate the whitelist with the following command:

     umi_tools whitelist --stdin hgmm_100_R1.fastq.gz \
                         --bc-pattern=CCCCCCCCCCCCCCCCNNNNNNNNNN \
                         --expect-cells=200 \
                         --plot-prefix=expect_whitelist \
                         --log2stderr > whitelist.txt

We can check how many CBs we accepted:

```
$ wc -l whitelist.txt
119 whitelist.txt
```

Based on our prior knowledge that there were ~100 true CBs, we've probably slightly over-estimated the number of CBs here. We can visualise this by looking at the following plots which present the same data in different ways to help assess the threshold:

![Counts distribution](./cell_barcode_count_density.png)
Figure 1. "expect_whitelist_cell_barcode_counts_density.png"
Distribution of read counts per CB. The least abundant CBs are removed from this plot

---

![Cumulative counts](./cell_barcode_knee.png)
Figure 2. "expect_whitelist_cell_barcode_knee.png"
Cumulative counts as number of accepted barcodes is increased

---

![Counts per barcode](./cell_barcode_counts.png)
Figure 3. "expect_whitelist_cell_barcode_counts.png"
Logs counts per barcode (ranked)

---

For all plots, the 'Selected' local minima (or threshold) is shown by the dashed line. If we had other local minima which were 'Rejected' these would also be shown. The thresholds are also tabulated in the file `expect_whitelist_cell_thresholds.tsv`.

In Figure 1, we can see two clear peaks, with the right peak representing the true cell barcodes.  In this case, the selected local minima appears to be in the correct place. This is confirmed in Figures 2 & 3, where the threshold appears to be at the 'knee' in the cumulative counts, and counts per barcode. So it looks like in this case, the automatic detection of the 'knee' has worked well. However, in some cases, the automatic selection may not work so well. In these case we can re-run `whitelist` and set the number of cells with the `--set-cell-number` option as described above.

In most cases, there will be many local minima and sometimes the wrong one may be selected. If so, you can insepect the `[PLOT_PREFIX]_cell_thresholds.tsv` file and re-run the `whitelist` command using the `--set-cell-number` option with the threshold which you have deemed to be most appropriate. We've not yet come across a case where no local minima identified near the 'knee', but if so, you can inspect the counts per barcode and cumulative counts figures and estimate a number to use with the `--set-cell-number` option.

### Correction of cell barcodes
In [Step 2](#Step-2:-Identifying-the-real-cells) above, we generated a whitelist of accepted ("true") CBs. By default, we also identified the CBs that were not accepted but were a single edit from a just one accepted barcode (column 2) in `whitelist.txt`. These are likely to be errors and can be corrected. By default, `extract` will only retain reads where the CB matches the whitelist. However, we can also accept these "error" CBs and correct the sequence to the accepted CB using the `--error-correct-cell` option.

### Quality filtering and masking barcodes
In some cases, the UMI bases may have very low base call quality scores. There are three options to deal with these reads:

1. Retain
2. Discard
3. Mask

By defualt, `extract` takes option 1 and does not considered quality scores. If you want to take option 2 and discard these reads you can do so with `--quality-filter-threshold=[FILTER]`, whereby all reads where the UMI contains a base with a Phred scrore below `FILTER` are discarded. However, if you are concerned about low quality bases, our recommendation is that you take option 3 and use `--quality-filter-mask=[FILTER]` to replace all UMI bases with a Phred score below `FILTER` with an "N". UMIs with an "N" will be 1-edit distance away from their cognate error-free UMI sequence and will therefore be handled sensibly by the downstream UMI-tools commands.

### Going back a step in the group>dedup>count proceedure

In this guide we have discussed the the UMI-tools `whitelist`, `extract` and `count` commands. There are two further UMI-tools commands which deal with BAM files, `group` and `dedup`.  `group`, `dedup` and `count` can be thought of as performing sequential levels of analysis in the process of obtaining read counts per gene: First the UMIs are "grouped" according to the gene to which they are assigned, then the UMIs are "deduplicated" to leave a single UMI per group, then the UMIs per gene are "counted". In some cases, you may wish to stop the proceedure prior to counting to inspect the grouping or deduplication process. This can be easily achieved by simply replacing the `count` command with `dedup` or `group`, with a very few considerations to command specific options:

* `--wide-format-cell-counts` is a `count`-specific option
* When running `dedup`, you may want to include the `--output-stats` option
* To output a BAM with `group`, you need to supply the `--output-bam` option

**warning**:
Running the `group` command with the `--per-gene` option in order to assess the groups used in the process of counting is very memory intensive, especially when the reads have been aligned to a genome, since all reads for a given contig in the BAM must be kept in memory as python objects until the whole contig has been parsed. This is not the case for `count` since we only retain the tally of reads. We recommend if you want to `group` together with `--per-gene` you used a transcriptome mapping (see below)

### Mapping to the transcriptome rather than genome

Many single cell protocols recommend mapping to the transcriptome rather than then genome. This makes downstream processing easier. In paritcular it makes it easier to know when you've seen all the reads from a gene, so deduplication can happen. This leads to a reduced memory footprint, which is particularly obvious in `dedup` and `group`. However, there are problems.

1. When you map to the transcriptome almost all reads are multimapping as most sequence in a gene is part of more than one transcript. This makes it difficult to distinguish between genuninely multimapping reads and reads that just come from multiple transcripts of the same gene.
2. If we include these reads we also face the problem that reads will be double counted, and each time the deduplication is done, it will be done using a different set of other reads, thus a read might filtered as a duplicate for one transcript but not another.
3. Reads that should map elsewhere might be forced to map to the transcriptome.

It turns out we can solve either problems 1 + 2 or problem 3. This is why we recommend mapping to the genome and annotating with `featureCounts` above.

**Method 1**
We solve problems 1 and 2 by using a collapsed transcriptome where all transcripts for a gene are merged into a single model. This is similar to the superTranscript concept proposed by [Davidison *et al*](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-017-1284-1). You can download our merged transcriptome annotations for human GENCODE 26 and mouse GENCODE 15 from ==here and here==, or see below for instructions on generating your own. You should generate FASTA from these, and then genome indexes for your favorite spliced aligner. After alignment the contigs of the aligned BAM file will contain the gene_id. You can then run `count`, `dedup` or `group` with:

```
$ umi_tools [COMMAND] --per-contig -I aligned.bam
```

You'll find that after the effort of generating the genome and its indexes, the `umi_tools` step is quicker and uses less memory, and the `featureCounts` step is unecessary.

**Method 2**
The above doesn't solve the third problem above however. We can get transcriptome aligned reads without the problem three using `STAR`. STAR has an output mode `--quantMode TranscriptomeSAM` where reads are mapped to the genome, but then their mapping coordinates are translated to the transcriptome and output in that form. For this you would pass `STAR` a normal transcriptome (i.e. not one of the collasped ones from above) using `--sjdbGTFfile` option. You could then sort the `BAM` file and run `umi_tools` with:

```
$ umi_tools [COMMAND] --per-contig --per-gene --gene-transcript-map=[MAPFILE]
```

where `MAPFILE` is a tab-sepearted text file mapping transcript_ids (in column 1) to gene_ids (in column 2).

However, since we might encounter the same read twice (mapped to different transcripts of the same gene), UMI counts might be inaccurate.

Ideally we'd like to combine these two methods: map to the genome and then have STAR translate to merged transcriptome coordinates. Unfortunately `STAR` will only transfer reads that fit the intron/exon structure of the collapsed gene, which looses many splice junctions, and therefore about 10% of reads are lost.

### Generating merge transcriptome annotations

We generated the above linked collapsed transcriptomes using the [`cgat` pacakge](https://github.com/CGATOxford/cgat) and files from gencode. This involves repeated use of the `gtf2gtf` tool, which first merges the transcripts, and then sets the `transcript_id` of every exon to be equal to the `gene_id`.

For example, for the human transcriptome:

```
$ wget ftp://ftp.sanger.ac.uk/pub/gencode/Gencode_human/release_26/gencode.v26.annotation.gtf.gz
$   cgat gtf2gtf --method=merge-exons -I gencode.v26.annotation.gtf.gz \
  | cgat gtf2gtf --method=set-transcript-to-gene -S hg38_genocode26_merged.gtf.gz
```

[1]: http://www.sciencedirect.com/science/article/pii/S0092867415005498
[2]: https://www.ncbi.nlm.nih.gov/pubmed/26000488


text:
/**
 * Client module
 *
 * This module contains the factory method for creating Client classes, and the
 * method calling code for all types of methods.
 *
 * @example <caption>Create a client and call a method on it</caption>
 *
 * var proto_obj = grpc.load(proto_file_path);
 * var Client = proto_obj.package.subpackage.ServiceName;
 * var client = new Client(server_address, client_credentials);
 * var call = client.unaryMethod(arguments, callback);
 */

'use strict';

var _ = require('lodash');

var client_interceptors = require('./client_interceptors');
var grpc = require('./grpc_extension');

var common = require('./common');

var Metadata = require('./metadata');

var constants = require('./constants');

var EventEmitter = require('events').EventEmitter;

var stream = require('stream');

var Readable = stream.Readable;
var Writable = stream.Writable;
var Duplex = stream.Duplex;
var methodTypes = constants.methodTypes;
var util = require('util');
var version = require('../package.json').version;

/**
 * Initial response metadata sent by the server when it starts processing the
 * call
 * @event grpc~ClientUnaryCall#metadata
 * @type {grpc.Metadata}
 */

/**
 * Status of the call when it has completed.
 * @event grpc~ClientUnaryCall#status
 * @type grpc~StatusObject
 */

util.inherits(ClientUnaryCall, EventEmitter);

/**
 * An EventEmitter. Used for unary calls.
 * @constructor grpc~ClientUnaryCall
 * @extends external:EventEmitter
 * @param {grpc.internal~Call} call The call object associated with the request
 */
function ClientUnaryCall(call) {
  EventEmitter.call(this);
  this.call = call;
}

util.inherits(ClientWritableStream, Writable);

/**
 * A stream that the client can write to. Used for calls that are streaming from
 * the client side.
 * @constructor grpc~ClientWritableStream
 * @extends external:Writable
 * @borrows grpc~ClientUnaryCall#cancel as grpc~ClientWritableStream#cancel
 * @borrows grpc~ClientUnaryCall#getPeer as grpc~ClientWritableStream#getPeer
 * @borrows grpc~ClientUnaryCall#event:metadata as
 *     grpc~ClientWritableStream#metadata
 * @borrows grpc~ClientUnaryCall#event:status as
 *     grpc~ClientWritableStream#status
 * @param {InterceptingCall} call Exposes gRPC request operations, processed by
 *     an interceptor stack.
 */
function ClientWritableStream(call) {
  Writable.call(this, {objectMode: true});
  this.call = call;
  var self = this;
  this.on('finish', function() {
    self.call.halfClose();
  });
}

/**
 * Write a message to the request stream. If serializing the argument fails,
 * the call will be cancelled and the stream will end with an error.
 * @name grpc~ClientWritableStream#write
 * @kind function
 * @override
 * @param {*} message The message to write. Must be a valid argument to the
 *     serialize function of the corresponding method
 * @param {grpc.writeFlags} flags Flags to modify how the message is written
 * @param {Function} callback Callback for when this chunk of data is flushed
 * @return {boolean} As defined for [Writable]{@link external:Writable}
 */

/**
 * Attempt to write the given chunk. Calls the callback when done. This is an
 * implementation of a method needed for implementing stream.Writable.
 * @private
 * @param {*} chunk The chunk to write
 * @param {grpc.writeFlags} encoding Used to pass write flags
 * @param {function(Error=)} callback Called when the write is complete
 */
function _write(chunk, encoding, callback) {
  /* jshint validthis: true */
  var self = this;
  if (this.writeFailed) {
    /* Once a write fails, just call the callback immediately to let the caller
       flush any pending writes. */
    setImmediate(callback);
    return;
  }
  var outerCallback = function(err, event) {
    if (err) {
      /* Assume that the call is complete and that writing failed because a
         status was received. In that case, set a flag to discard all future
         writes */
      self.writeFailed = true;
    }
    callback();
  };
  var context = {
    encoding: encoding,
    callback: outerCallback
  };
  this.call.sendMessageWithContext(context, chunk);
}

ClientWritableStream.prototype._write = _write;

util.inherits(ClientReadableStream, Readable);

/**
 * A stream that the client can read from. Used for calls that are streaming
 * from the server side.
 * @constructor grpc~ClientReadableStream
 * @extends external:Readable
 * @borrows grpc~ClientUnaryCall#cancel as grpc~ClientReadableStream#cancel
 * @borrows grpc~ClientUnaryCall#getPeer as grpc~ClientReadableStream#getPeer
 * @borrows grpc~ClientUnaryCall#event:metadata as
 *     grpc~ClientReadableStream#metadata
 * @borrows grpc~ClientUnaryCall#event:status as
 *     grpc~ClientReadableStream#status
 * @param {InterceptingCall} call Exposes gRPC request operations, processed by
 *     an interceptor stack.
 */
function ClientReadableStream(call) {
  Readable.call(this, {objectMode: true});
  this.call = call;
  this.finished = false;
  this.reading = false;
  /* Status generated from reading messages from the server. Overrides the
   * status from the server if not OK */
  this.read_status = null;
  /* Status received from the server. */
  this.received_status = null;
}

/**
 * Called when all messages from the server have been processed. The status
 * parameter indicates that the call should end with that status. status
 * defaults to OK if not provided.
 * @param {Object!} status The status that the call should end with
 * @private
 */
function _readsDone(status) {
  /* jshint validthis: true */
  if (!status) {
    status = {code: constants.status.OK, details: 'OK'};
  }
  if (status.code !== constants.status.OK) {
    this.call.cancelWithStatus(status.code, status.details);
  }
  this.finished = true;
  this.read_status = status;
  this._emitStatusIfDone();
}

ClientReadableStream.prototype._readsDone = _readsDone;

/**
 * Called to indicate that we have received a status from the server.
 * @private
 */
function _receiveStatus(status) {
  /* jshint validthis: true */
  this.received_status = status;
  this._emitStatusIfDone();
}

ClientReadableStream.prototype._receiveStatus = _receiveStatus;

/**
 * If we have both processed all incoming messages and received the status from
 * the server, emit the status. Otherwise, do nothing.
 * @private
 */
function _emitStatusIfDone() {
  /* jshint validthis: true */
  var status;
  if (this.read_status && this.received_status) {
    if (this.read_status.code !== constants.status.OK) {
      status = this.read_status;
    } else {
      status = this.received_status;
    }
    if (status.code === constants.status.OK) {
      this.push(null);
    } else {
      var error = common.createStatusError(status);
      this.emit('error', error);
    }
    this.emit('status', status);
  }
}

ClientReadableStream.prototype._emitStatusIfDone = _emitStatusIfDone;

/**
 * Read the next object from the stream.
 * @private
 * @param {*} size Ignored because we use objectMode=true
 */
function _read(size) {
  /* jshint validthis: true */
  if (this.finished) {
    this.push(null);
  } else {
    if (!this.reading) {
      this.reading = true;
      var context = {
        stream: this
      };
      this.call.recvMessageWithContext(context);
    }
  }
}

ClientReadableStream.prototype._read = _read;

util.inherits(ClientDuplexStream, Duplex);

/**
 * A stream that the client can read from or write to. Used for calls with
 * duplex streaming.
 * @constructor grpc~ClientDuplexStream
 * @extends external:Duplex
 * @borrows grpc~ClientUnaryCall#cancel as grpc~ClientDuplexStream#cancel
 * @borrows grpc~ClientUnaryCall#getPeer as grpc~ClientDuplexStream#getPeer
 * @borrows grpc~ClientWritableStream#write as grpc~ClientDuplexStream#write
 * @borrows grpc~ClientUnaryCall#event:metadata as
 *     grpc~ClientDuplexStream#metadata
 * @borrows grpc~ClientUnaryCall#event:status as
 *     grpc~ClientDuplexStream#status
 * @param {InterceptingCall} call Exposes gRPC request operations, processed by
 *     an interceptor stack.
 */
function ClientDuplexStream(call) {
  Duplex.call(this, {objectMode: true});
  this.call = call;
  /* Status generated from reading messages from the server. Overrides the
   * status from the server if not OK */
  this.read_status = null;
  /* Status received from the server. */
  this.received_status = null;
  var self = this;
  this.on('finish', function() {
    self.call.halfClose();
  });
}

ClientDuplexStream.prototype._readsDone = _readsDone;
ClientDuplexStream.prototype._receiveStatus = _receiveStatus;
ClientDuplexStream.prototype._emitStatusIfDone = _emitStatusIfDone;
ClientDuplexStream.prototype._read = _read;
ClientDuplexStream.prototype._write = _write;

/**
 * Cancel the ongoing call. Results in the call ending with a CANCELLED status,
 * unless it has already ended with some other status.
 * @alias grpc~ClientUnaryCall#cancel
 */
function cancel() {
  /* jshint validthis: true */
  this.call.cancel();
}

ClientUnaryCall.prototype.cancel = cancel;
ClientReadableStream.prototype.cancel = cancel;
ClientWritableStream.prototype.cancel = cancel;
ClientDuplexStream.prototype.cancel = cancel;

/**
 * Get the endpoint this call/stream is connected to.
 * @return {string} The URI of the endpoint
 * @alias grpc~ClientUnaryCall#getPeer
 */
function getPeer() {
  /* jshint validthis: true */
  return this.call.getPeer();
}

ClientUnaryCall.prototype.getPeer = getPeer;
ClientReadableStream.prototype.getPeer = getPeer;
ClientWritableStream.prototype.getPeer = getPeer;
ClientDuplexStream.prototype.getPeer = getPeer;

/**
 * Any client call type
 * @typedef {(grpc~ClientUnaryCall|grpc~ClientReadableStream|
 *            grpc~ClientWritableStream|grpc~ClientDuplexStream)}
 *     grpc.Client~Call
 */

/**
 * Options that can be set on a call.
 * @typedef {Object} grpc.Client~CallOptions
 * @property {grpc~Deadline} deadline The deadline for the entire call to
 *     complete.
 * @property {string} host Server hostname to set on the call. Only meaningful
 *     if different from the server address used to construct the client.
 * @property {grpc.Client~Call} parent Parent call. Used in servers when
 *     making a call as part of the process of handling a call. Used to
 *     propagate some information automatically, as specified by
 *     propagate_flags.
 * @property {number} propagate_flags Indicates which properties of a parent
 *     call should propagate to this call. Bitwise combination of flags in
 *     {@link grpc.propagate}.
 * @property {grpc.credentials~CallCredentials} credentials The credentials that
 *     should be used to make this particular call.
 */

/**
 * A generic gRPC client. Primarily useful as a base class for generated clients
 * @memberof grpc
 * @constructor
 * @param {string} address Server address to connect to
 * @param {grpc.credentials~ChannelCredentials} credentials Credentials to use
 *     to connect to the server
 * @param {Object} options Options to apply to channel creation
 */
function Client(address, credentials, options) {
  var self = this;
  if (!options) {
    options = {};
  }
  /* Append the grpc-node user agent string after the application user agent
   * string, and put the combination at the beginning of the user agent string
   */
  if (options['grpc.primary_user_agent']) {
    options['grpc.primary_user_agent'] += ' ';
  } else {
    options['grpc.primary_user_agent'] = '';
  }
  options['grpc.primary_user_agent'] += 'grpc-node/' + version;

  // Resolve interceptor options and assign interceptors to each method
  var interceptor_providers = options.interceptor_providers || [];
  var interceptors = options.interceptors || [];
  if (interceptor_providers.length && interceptors.length) {
    throw new client_interceptors.InterceptorConfigurationError(
      'Both interceptors and interceptor_providers were passed as options ' +
      'to the client constructor. Only one of these is allowed.');
  }
  _.each(self.$method_definitions, function(method_definition, method_name) {
    self[method_name].interceptors = client_interceptors
      .resolveInterceptorProviders(interceptor_providers, method_definition)
      .concat(interceptors);
  });

  // Exclude interceptor options which have already been consumed
  var channel_options = _.omit(options,
     ['interceptors', 'interceptor_providers']);
  /* Private fields use $ as a prefix instead of _ because it is an invalid
   * prefix of a method name */
  this.$channel = new grpc.Channel(address, credentials, channel_options);
}

exports.Client = Client;

/**
 * @callback grpc.Client~requestCallback
 * @param {?grpc~ServiceError} error The error, if the call
 *     failed
 * @param {*} value The response value, if the call succeeded
 */

/**
 * Make a unary request to the given method, using the given serialize
 * and deserialize functions, with the given argument.
 * @param {string} path The path of the method to request
 * @param {grpc~serialize} serialize The serialization function for
 *     inputs
 * @param {grpc~deserialize} deserialize The deserialization
 *     function for outputs
 * @param {*} argument The argument to the call. Should be serializable with
 *     serialize
 * @param {grpc.Metadata=} metadata Metadata to add to the call
 * @param {grpc.Client~CallOptions=} options Options map
 * @param {grpc.Client~requestCallback} callback The callback
 *     for when the response is received
 * @return {grpc~ClientUnaryCall} An event emitter for stream related events
 */
Client.prototype.makeUnaryRequest = function(path, serialize, deserialize,
                                             argument, metadata, options,
                                             callback) {
  if (_.isFunction(options)) {
    callback = options;
    if (metadata instanceof Metadata) {
      options = {};
    } else {
      options = metadata;
      metadata = new Metadata();
    }
  } else if (_.isFunction(metadata)) {
    callback = metadata;
    metadata = new Metadata();
    options = {};
  }
  if (!metadata) {
    metadata = new Metadata();
  }
  if (!options) {
    options = {};
  }
  if (!((metadata instanceof Metadata) &&
        (options instanceof Object) &&
        (_.isFunction(callback)))) {
    throw new Error('Argument mismatch in makeUnaryRequest');
  }

  var method_name = this.$method_names[path];
  var constructor_interceptors = this[method_name] ?
    this[method_name].interceptors :
    null;
  var method_definition = options.method_definition = {
    path: path,
    requestStream: false,
    responseStream: false,
    requestSerialize: serialize,
    responseDeserialize: deserialize
  };

  metadata = metadata.clone();

  var intercepting_call = client_interceptors.getInterceptingCall(
    method_definition,
    options,
    constructor_interceptors,
    this.$channel,
    callback
  );
  var emitter = new ClientUnaryCall(intercepting_call);
  var last_listener = client_interceptors.getLastListener(
    method_definition,
    emitter,
    callback
  );

  intercepting_call.start(metadata, last_listener);
  intercepting_call.sendMessage(argument);
  intercepting_call.halfClose();

  return emitter;
};

/**
 * Make a client stream request to the given method, using the given serialize
 * and deserialize functions, with the given argument.
 * @param {string} path The path of the method to request
 * @param {grpc~serialize} serialize The serialization function for
 *     inputs
 * @param {grpc~deserialize} deserialize The deserialization
 *     function for outputs
 * @param {grpc.Metadata=} metadata Array of metadata key/value pairs to add to
 *     the call
 * @param {grpc.Client~CallOptions=} options Options map
 * @param {grpc.Client~requestCallback} callback The callback for when the
 *     response is received
 * @return {grpc~ClientWritableStream} An event emitter for stream related
 *     events
 */
Client.prototype.makeClientStreamRequest = function(path, serialize,
                                                    deserialize, metadata,
                                                    options, callback) {
  if (_.isFunction(options)) {
    callback = options;
    if (metadata instanceof Metadata) {
      options = {};
    } else {
      options = metadata;
      metadata = new Metadata();
    }
  } else if (_.isFunction(metadata)) {
    callback = metadata;
    metadata = new Metadata();
    options = {};
  }
  if (!metadata) {
    metadata = new Metadata();
  }
  if (!options) {
    options = {};
  }
  if (!((metadata instanceof Metadata) &&
       (options instanceof Object) &&
       (_.isFunction(callback)))) {
    throw new Error('Argument mismatch in makeClientStreamRequest');
  }

  var method_name = this.$method_names[path];
  var constructor_interceptors = this[method_name] ?
    this[method_name].interceptors :
    null;
  var method_definition = options.method_definition = {
    path: path,
    requestStream: true,
    responseStream: false,
    requestSerialize: serialize,
    responseDeserialize: deserialize
  };

  metadata = metadata.clone();

  var intercepting_call = client_interceptors.getInterceptingCall(
    method_definition,
    options,
    constructor_interceptors,
    this.$channel,
    callback
  );
  var emitter = new ClientWritableStream(intercepting_call);
  var last_listener = client_interceptors.getLastListener(
    method_definition,
    emitter,
    callback
  );

  intercepting_call.start(metadata, last_listener);

  return emitter;
};

/**
 * Make a server stream request to the given method, with the given serialize
 * and deserialize function, using the given argument
 * @param {string} path The path of the method to request
 * @param {grpc~serialize} serialize The serialization function for inputs
 * @param {grpc~deserialize} deserialize The deserialization
 *     function for outputs
 * @param {*} argument The argument to the call. Should be serializable with
 *     serialize
 * @param {grpc.Metadata=} metadata Array of metadata key/value pairs to add to
 *     the call
 * @param {grpc.Client~CallOptions=} options Options map
 * @return {grpc~ClientReadableStream} An event emitter for stream related
 *     events
 */
Client.prototype.makeServerStreamRequest = function(path, serialize,
                                                    deserialize, argument,
                                                    metadata, options) {
  if (!(metadata instanceof Metadata)) {
    options = metadata;
    metadata = new Metadata();
  }
  if (!(options instanceof Object)) {
    options = {};
  }
  if (!((metadata instanceof Metadata) && (options instanceof Object))) {
    throw new Error('Argument mismatch in makeServerStreamRequest');
  }

  var method_name = this.$method_names[path];
  var constructor_interceptors = this[method_name] ?
    this[method_name].interceptors :
    null;
  var method_definition = options.method_definition = {
    path: path,
    requestStream: false,
    responseStream: true,
    requestSerialize: serialize,
    responseDeserialize: deserialize
  };

  metadata = metadata.clone();

  var emitter = new ClientReadableStream();
  var intercepting_call = client_interceptors.getInterceptingCall(
    method_definition,
    options,
    constructor_interceptors,
    this.$channel,
    emitter
  );
  emitter.call = intercepting_call;
  var last_listener = client_interceptors.getLastListener(
    method_definition,
    emitter
  );

  intercepting_call.start(metadata, last_listener);
  intercepting_call.sendMessage(argument);
  intercepting_call.halfClose();

  return emitter;
};

/**
 * Make a bidirectional stream request with this method on the given channel.
 * @param {string} path The path of the method to request
 * @param {grpc~serialize} serialize The serialization function for inputs
 * @param {grpc~deserialize} deserialize The deserialization
 *     function for outputs
 * @param {grpc.Metadata=} metadata Array of metadata key/value
 *     pairs to add to the call
 * @param {grpc.Client~CallOptions=} options Options map
 * @return {grpc~ClientDuplexStream} An event emitter for stream related events
 */
Client.prototype.makeBidiStreamRequest = function(path, serialize,
                                                  deserialize, metadata,
                                                  options) {
  if (!(metadata instanceof Metadata)) {
    options = metadata;
    metadata = new Metadata();
  }
  if (!(options instanceof Object)) {
    options = {};
  }
  if (!((metadata instanceof Metadata) && (options instanceof Object))) {
    throw new Error('Argument mismatch in makeBidiStreamRequest');
  }

  var method_name = this.$method_names[path];
  var constructor_interceptors = this[method_name] ?
    this[method_name].interceptors :
    null;
  var method_definition = options.method_definition = {
    path: path,
    requestStream: true,
    responseStream: true,
    requestSerialize: serialize,
    responseDeserialize: deserialize
  };

  metadata = metadata.clone();

  var emitter = new ClientDuplexStream();
  var intercepting_call = client_interceptors.getInterceptingCall(
    method_definition,
    options,
    constructor_interceptors,
    this.$channel,
    emitter
  );
  emitter.call = intercepting_call;
  var last_listener = client_interceptors.getLastListener(
    method_definition,
    emitter
  );

  intercepting_call.start(metadata, last_listener);

  return emitter;
};

/**
 * Close this client.
 */
Client.prototype.close = function() {
  this.$channel.close();
};

/**
 * Return the underlying channel object for the specified client
 * @return {Channel} The channel
 */
Client.prototype.getChannel = function() {
  return this.$channel;
};

/**
 * Wait for the client to be ready. The callback will be called when the
 * client has successfully connected to the server, and it will be called
 * with an error if the attempt to connect to the server has unrecoverablly
 * failed or if the deadline expires. This function will make the channel
 * start connecting if it has not already done so.
 * @param {grpc~Deadline} deadline When to stop waiting for a connection.
 * @param {function(Error)} callback The callback to call when done attempting
 *     to connect.
 */
Client.prototype.waitForReady = function(deadline, callback) {
  var self = this;
  var checkState = function(err) {
    if (err) {
      callback(new Error('Failed to connect before the deadline'));
      return;
    }
    var new_state = self.$channel.getConnectivityState(true);
    if (new_state === grpc.connectivityState.READY) {
      callback();
    } else if (new_state === grpc.connectivityState.FATAL_FAILURE) {
      callback(new Error('Failed to connect to server'));
    } else {
      self.$channel.watchConnectivityState(new_state, deadline, checkState);
    }
  };
  /* Force a single round of polling to ensure that the channel state is up
   * to date */
  grpc.forcePoll();
  setImmediate(checkState);
};

/**
 * Map with short names for each of the requester maker functions. Used in
 * makeClientConstructor
 * @private
 */
var requester_funcs = {
  [methodTypes.UNARY]: Client.prototype.makeUnaryRequest,
  [methodTypes.CLIENT_STREAMING]: Client.prototype.makeClientStreamRequest,
  [methodTypes.SERVER_STREAMING]: Client.prototype.makeServerStreamRequest,
  [methodTypes.BIDI_STREAMING]: Client.prototype.makeBidiStreamRequest
};

function getDefaultValues(metadata, options) {
  var res = {};
  res.metadata = metadata || new Metadata();
  res.options = options || {};
  return res;
}

/**
 * Map with wrappers for each type of requester function to make it use the old
 * argument order with optional arguments after the callback.
 * @access private
 */
var deprecated_request_wrap = {
  [methodTypes.UNARY]: function(makeUnaryRequest) {
    return function makeWrappedUnaryRequest(argument, callback,
                                            metadata, options) {
      /* jshint validthis: true */
      var opt_args = getDefaultValues(metadata, metadata);
      return makeUnaryRequest.call(this, argument, opt_args.metadata,
                                   opt_args.options, callback);
    };
  },
  [methodTypes.CLIENT_STREAMING]: function(makeServerStreamRequest) {
    return function makeWrappedClientStreamRequest(callback, metadata,
                                                   options) {
      /* jshint validthis: true */
      var opt_args = getDefaultValues(metadata, options);
      return makeServerStreamRequest.call(this, opt_args.metadata,
                                          opt_args.options, callback);
    };
  },
  [methodTypes.SERVER_STREAMING]: _.identity,
  [methodTypes.BIDI_STREAMING]: _.identity
};

/**
 * Creates a constructor for a client with the given methods, as specified in
 * the methods argument. The resulting class will have an instance method for
 * each method in the service, which is a partial application of one of the
 * [Client]{@link grpc.Client} request methods, depending on `requestSerialize`
 * and `responseSerialize`, with the `method`, `serialize`, and `deserialize`
 * arguments predefined.
 * @memberof grpc
 * @alias grpc~makeGenericClientConstructor
 * @param {grpc~ServiceDefinition} methods An object mapping method names to
 *     method attributes
 * @param {string} serviceName The fully qualified name of the service
 * @param {Object} class_options An options object.
 * @param {boolean=} [class_options.deprecatedArgumentOrder=false] Indicates
 *     that the old argument order should be used for methods, with optional
 *     arguments at the end instead of the callback at the end. This option
 *     is only a temporary stopgap measure to smooth an API breakage.
 *     It is deprecated, and new code should not use it.
 * @return {function} New client constructor, which is a subclass of
 *     {@link grpc.Client}, and has the same arguments as that constructor.
 */
exports.makeClientConstructor = function(methods, serviceName,
                                         class_options) {
  if (!class_options) {
    class_options = {};
  }

  function ServiceClient(address, credentials, options) {
    Client.call(this, address, credentials, options);
  }

  util.inherits(ServiceClient, Client);
  ServiceClient.prototype.$method_definitions = methods;
  ServiceClient.prototype.$method_names = {};

  _.each(methods, function(attrs, name) {
    if (_.startsWith(name, '$')) {
      throw new Error('Method names cannot start with $');
    }
    var method_type = common.getMethodType(attrs);
    var method_func = _.partial(requester_funcs[method_type], attrs.path,
                                attrs.requestSerialize,
                                attrs.responseDeserialize);
    if (class_options.deprecatedArgumentOrder) {
      ServiceClient.prototype[name] =
        deprecated_request_wrap[method_type](method_func);
    } else {
      ServiceClient.prototype[name] = method_func;
    }
    ServiceClient.prototype.$method_names[attrs.path] = name;
    // Associate all provided attributes with the method
    _.assign(ServiceClient.prototype[name], attrs);
    if (attrs.originalName) {
      ServiceClient.prototype[attrs.originalName] =
        ServiceClient.prototype[name];
    }
  });

  ServiceClient.service = methods;

  return ServiceClient;
};

/**
 * Return the underlying channel object for the specified client
 * @memberof grpc
 * @alias grpc~getClientChannel
 * @param {grpc.Client} client The client
 * @return {Channel} The channel
 * @see grpc.Client#getChannel
 */
exports.getClientChannel = function(client) {
  return Client.prototype.getChannel.call(client);
};

/**
 * Gets a map of client method names to interceptor stacks.
 * @param {grpc.Client} client
 * @returns {Object.<string, Interceptor[]>}
 */
exports.getClientInterceptors = function(client) {
  return _.mapValues(client.$method_definitions, function(def, name) {
    return client[name].interceptors;
  });
};

/**
 * Wait for the client to be ready. The callback will be called when the
 * client has successfully connected to the server, and it will be called
 * with an error if the attempt to connect to the server has unrecoverablly
 * failed or if the deadline expires. This function will make the channel
 * start connecting if it has not already done so.
 * @memberof grpc
 * @alias grpc~waitForClientReady
 * @param {grpc.Client} client The client to wait on
 * @param {grpc~Deadline} deadline When to stop waiting for a connection. Pass
 *     Infinity to wait forever.
 * @param {function(Error)} callback The callback to call when done attempting
 *     to connect.
 * @see grpc.Client#waitForReady
 */
exports.waitForClientReady = function(client, deadline, callback) {
  Client.prototype.waitForReady.call(client, deadline, callback);
};

exports.StatusBuilder = client_interceptors.StatusBuilder;
exports.ListenerBuilder = client_interceptors.ListenerBuilder;
exports.RequesterBuilder = client_interceptors.RequesterBuilder;
exports.InterceptingCall = client_interceptors.InterceptingCall;

redpajama_set_name: RedPajamaGithub


text:
'use strict';

module.exports = function(app) {
	var users = require('../../app/controllers/users.server.controller');
	var skills = require('../../app/controllers/skills.server.controller');

	// Skills Routes
	app.route('/skills')
		.get(skills.list)
		.post(users.requiresLogin, skills.create);

	app.route('/skills/:skillId')
		.get(skills.read)
		.put(users.requiresLogin, skills.hasAuthorization, skills.update)
		.delete(users.requiresLogin, skills.hasAuthorization, skills.delete);

	// Finish by binding the Skill middleware
	app.param('skillId', skills.skillByID);
};


text:
# Checkboxes

The MDC Checkbox component is a spec-aligned checkbox component adhering to the
[Material Design checkbox requirements](https://material.io/guidelines/components/selection-controls.html#selection-controls-checkbox).
It works without JavaScript with basic functionality for all states. If you use the JavaScript object for a checkbox, it will add more intricate animation effects when switching between states.

## Design & API Documentation

    Material Design guidelines: Selection Controls – Checkbox

    Demo

## Installation

```
npm install --save @material/checkbox
```

## Usage

### Standalone Checkbox

```html
<div class="mdc-checkbox">
  <input type="checkbox"
         class="mdc-checkbox__native-control"/>
  <div class="mdc-checkbox__background">
    <svg class="mdc-checkbox__checkmark"
         viewBox="0 0 24 24">
      <path class="mdc-checkbox__checkmark__path"
            fill="none"
            stroke="white"
            d="M1.73,12.91 8.1,19.28 22.79,4.59"/>
    </svg>
    <div class="mdc-checkbox__mixedmark"></div>
  </div>
</div>
```

The checkbox component is driven by an underlying native checkbox element. This element is sized and
positioned the same way as the checkbox component itself, allowing for proper behavior of assistive
devices.

You can also add an `mdc-checkbox--theme-dark` modifier class to the component to use the dark theme
checkbox styles.

Additionally, the checkbox can be used in conjunction with [mdc-form-field](../mdc-form-field) to
easily position checkboxes and their labels.

```html
<div class="mdc-form-field">
  <div class="mdc-checkbox">
    <input type="checkbox"
           id="my-checkbox"
           class="mdc-checkbox__native-control"/>
    <div class="mdc-checkbox__background">
      <svg class="mdc-checkbox__checkmark"
           viewBox="0 0 24 24">
        <path class="mdc-checkbox__checkmark__path"
              fill="none"
              stroke="white"
              d="M1.73,12.91 8.1,19.28 22.79,4.59"/>
      </svg>
      <div class="mdc-checkbox__mixedmark"></div>
    </div>
  </div>

  <label for="my-checkbox">My Checkbox Label</label>
</div>
```

#### Disabled Checkboxes

```html
<div class="mdc-checkbox mdc-checkbox--disabled">
  <input type="checkbox"
         id="basic-disabled-checkbox"
         class="mdc-checkbox__native-control"
         disabled />
  <div class="mdc-checkbox__background">
    <svg class="mdc-checkbox__checkmark"
         viewBox="0 0 24 24">
      <path class="mdc-checkbox__checkmark__path"
            fill="none"
            stroke="white"
            d="M1.73,12.91 8.1,19.28 22.79,4.59"/>
    </svg>
    <div class="mdc-checkbox__mixedmark"></div>
  </div>
</div>
<label for="basic-disabled-checkbox" id="basic-disabled-checkbox-label">This is my disabled checkbox</label>
```

Note that `mdc-checkbox--disabled` is necessary on the root element to prevent hover states from activating.

### Using the JS Component

MDC Checkbox ships with a Component / Foundation combo which progressively enhances the checkbox
state transitions to achieve full parity with the Material Design motion for switching checkbox
states.

#### Including in code

##### ES2015

```javascript
import {MDCCheckbox, MDCCheckboxFoundation} from 'mdc-checkbox';
```

##### CommonJS

```javascript
const mdcCheckbox = require('mdc-checkbox');
const MDCCheckbox = mdcCheckbox.MDCCheckbox;
const MDCCheckboxFoundation = mdcCheckbox.MDCCheckboxFoundation;
```

##### AMD

```javascript
require(['path/to/mdc-checkbox'], mdcCheckbox => {
  const MDCCheckbox = mdcCheckbox.MDCCheckbox;
  const MDCCheckboxFoundation = mdcCheckbox.MDCCheckboxFoundation;
});
```

##### Global

```javascript
const MDCCheckbox = mdc.checkbox.MDCCheckbox;
const MDCCheckboxFoundation = mdc.checkbox.MDCCheckboxFoundation;
```

#### Automatic Instantiation

If you do not care about retaining the component instance for the checkbox, simply call `attachTo()`
and pass it a DOM element.

```javascript
mdc.checkbox.MDCCheckbox.attachTo(document.querySelector('.mdc-checkbox'));
```

#### Manual Instantiation

Checkboxes can easily be initialized using their default constructors as well, similar to `attachTo`.

```javascript
import {MDCCheckbox} from 'mdc-checkbox';

const checkbox = new MDCCheckbox(document.querySelector('.mdc-checkbox'));
```

#### MDCCheckbox API

The MDCCheckbox API provides accessor properties similar to those found on a native checkbox element.

##### MDCCheckbox.checked

Boolean. Returns whether or not the checkbox is checked. Setting this property will update the
underlying checkbox element.

##### MDCCheckbox.indeterminate

Boolean. Returns whether or not the checkbox is indeterminate. Setting this property will update the
underlying checkbox element.

##### MDCCheckbox.disabled

Boolean. Returns whether or not the checkbox is disabled. Setting this property will update the
underlying checkbox element.

##### MDCCheckbox.value

String. Returns the checkbox's value. Setting this property will update the underlying checkbox
element.

### Using the Foundation Class

MDC Checkbox ships with an `MDCCheckboxFoundation` class that external frameworks and libraries can
use to integrate the component. As with all foundation classes, an adapter object must be provided.
The adapter for checkboxes must provide the following functions, with correct signatures:

| Method Signature | Description |
| --- | --- |
| `addClass(className: string) => void` | Adds a class to the root element. |
| `removeClass(className: string) => void` | Removes a class from the root element. |
| `registerAnimationEndHandler(handler: EventListener) => void` | Registers an event handler to be called when an `animationend` event is triggered on the root element. Note that you must account for vendor prefixes in order for this to work correctly. |
| `deregisterAnimationEndHandler(handler: EventListener) => void` | Deregisters an event handler from an `animationend` event listener. This will only be called with handlers that have previously been passed to `registerAnimationEndHandler` calls. |
| `registerChangeHandler(handler: EventListener) => void` | Registers an event handler to be called when a `change` event is triggered on the native control (_not_ the root element). |
| `deregisterChangeHandler(handler: EventListener) => void` | Deregisters an event handler that was previously passed to `registerChangeHandler`. |
| `getNativeControl() => HTMLInputElement?` | Returns the native checkbox control, if available. Note that if this control is not available, the methods that rely on it will exit gracefully.|
| `forceLayout() => void` | Force-trigger a layout on the root element. This is needed to restart animations correctly. If you find that you do not need to do this, you can simply make it a no-op. |
| `isAttachedToDOM() => boolean` | Returns true if the component is currently attached to the DOM, false otherwise.` |

#### MDCCheckboxFoundation API

##### MDCCheckboxFoundation.isChecked() => boolean

Returns whether or not the underlying input is checked. Returns false when no input is available.

##### MDCCheckboxFoundation.setChecked(checked: boolean)

Updates the `checked` property on the underlying input. Does nothing when the underlying input is
not present.

##### MDCCheckboxFoundation.isIndeterminate() => boolean

Returns whether or not the underlying input is indeterminate. Returns false when no input is
available.

##### MDCCheckboxFoundation.setIndeterminate(indeterminate: boolean)

Updates the `indeterminate` property on the underlying input. Does nothing when the underlying input
is not present.

##### MDCCheckboxFoundation.isDisabled() => boolean

Returns whether or not the underlying input is disabled. Returns false when no input is available.

##### MDCCheckboxFoundation.setDisabled(disabled: boolean)

Updates the `disabled` property on the underlying input. Does nothing when the underlying input is
not present.

##### MDCCheckboxFoundation.getValue() => string

Returns the value of `adapter.getNativeControl().value`. Returns `null` if `getNativeControl()`
does not return an object.

##### MDCCheckboxFoundation.setValue(value: string) => void

Sets the value of `adapter.getNativeControl().value`. Does nothing if `getNativeControl()` does
not return an object.

## Theming

> TK once mdc-theming lands.


text:
package controllers;

/**
 * Crud interface for District model.
 */
public class Districts extends CRUD {

}


text:
describe('UserListFactory', function() {
  var factory, httpBackend;

  beforeEach(function() {
    module('myApp');

    inject(function($httpBackend, UserListFactory) {
      factory = UserListFactory;
      httpBackend = $httpBackend;
      response = { "data" : [{"login" : "user1"}]}
      user = { "data" : "username"}
      httpBackend.when("GET", "https://api.github.com/orgs/makersacademy/members").respond(response.data);
      httpBackend.when("GET", "https://api.github.com/users/user1").respond(user);

    })
  })

  it('is an Object', function() {
    expect(factory).toBeNonEmptyObject();

  })

  it('has a method called #getList', function() {
    expect(factory.getList()).toBeDefined();
    expect(factory).toHaveMethod('getList');
  })

  it('sends a GET request to the github.api', function() {
       factory.getList();
       httpBackend.flush();
       expect(userArray[0].data).toEqual(user);
  })

});


text:
import pyxb
import po4
import address
import pyxb.binding.datatypes as xs

po = po4.purchaseOrder(orderDate=xs.date(1999, 10, 20))
po.shipTo = address.USAddress('Alice Smith', '123 Maple Street', 'Anytown', 'AK', 12341)
po.billTo = address.USAddress('Robert Smith', '8 Oak Avenue', 'Anytown', 'AK', 12341)

pyxb.RequireValidWhenGenerating(False)
print po.toxml("utf-8")


text:
package com.weebly.stevelosk.uv_buddy;

import android.content.Context;
import android.content.Intent;
import android.content.res.Resources;
import android.content.res.TypedArray;
import android.graphics.Color;
import android.net.Uri;
import android.support.annotation.ArrayRes;
import android.support.v7.app.AppCompatActivity;
import android.os.Bundle;
import android.support.v7.widget.Toolbar;
import android.view.Menu;
import android.view.MenuItem;
import android.view.View;
import android.widget.Button;
import android.widget.EditText;
import android.widget.TextView;

import java.util.Calendar;

public class MainActivity extends AppCompatActivity implements iAsyncCalling {

    private TextView resultTextView;
    private TextView UV_indexDescriptionTextView;
    private EditText enterZipCodeEditText;
    private Button getUV_withZipCodeButton;
    private Button getUV_withLocationButton;
    private Toolbar mActionBar;

    private Integer[] uviArray;

    private String TAG = "Main Activity";
    private Resources res;

    private String mZipCode;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        mActionBar = (Toolbar) findViewById(R.id.my_toolbar);
        setSupportActionBar(mActionBar);

        res = getResources();

        resultTextView = (TextView) findViewById(R.id.resultTextView);
        UV_indexDescriptionTextView = (TextView) findViewById(R.id.UV_indexDescriptionTextView);
        enterZipCodeEditText = (EditText) findViewById(R.id.zipCodeEditText);
        getUV_withZipCodeButton = (Button) findViewById(R.id.getUV_withZipCodeButton);
        //getUV_withLocationButton = (Button) findViewById(R.id.getUV_withLocationButton);

        mActionBar.inflateMenu(R.menu.menu);

        // Button Click listener
        getUV_withZipCodeButton.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View view) {

                MainActivity callingActivity = MainActivity.this;
                // logic flag for no reading
                Integer currentUV_index = -1;

                // pass in current zipCode and hour
                mZipCode = enterZipCodeEditText.getText().toString();

                // get current hour
                String hour = getTime();

                // commented out old async task /*
                /*
                GetUV_IndexAsync task = new GetUV_IndexAsync(
                        callingActivity, zipCode, hour);
                task.execute();
                */

                GetUV_IndexAsync2 task = new GetUV_IndexAsync2(
                        getApplicationContext(), callingActivity, mZipCode);
                task.execute();

            }
        });
    }

    protected void updateArray(Integer[] data) {
        // removed Log calls.  Not sure what this method is for anymore.
        this.uviArray = data;
        for (int i = 0; i < 24; i++) {

        }
    }

    public void update(Integer index) {

        if (index == -1) {
            // got here because there was an error
            reportErrorToUser();

        } else {
            try {
                resultTextView.setText(index.toString());
                // Get resources
                Resources res = getResources();

                // Get description
                String[] desc_array = res.getStringArray(R.array.UV_index_description_array);
                String description = desc_array[index];
                UV_indexDescriptionTextView.setText(description);

                // set background color
                updateColor(index, res);

            } catch (NullPointerException e) {
                reportErrorToUser();
            }
        }
    }

    @Override
    public boolean onCreateOptionsMenu(Menu menu) {
        getMenuInflater().inflate(R.menu.menu, menu);
        return true;
    }

    @Override
    public boolean onOptionsItemSelected(MenuItem item) {
        switch (item.getItemId()) {
            case R.id.showEpaScale:
                // User chose the "Settings" item, show the app settings UI...
                // Launch EPA scale

                String uriString = res.getString(R.string.epaScaleUri);
                Uri webUri = Uri.parse(uriString);
                Intent navToEpaIntent = new Intent(Intent.ACTION_VIEW);
                navToEpaIntent.setData(webUri);
                startActivity(navToEpaIntent);

                return true;

            case R.id.setAlarmIcon:
                Intent setAlarmActionIntent = new Intent(this, NewAlarmActivity.class);
                setAlarmActionIntent.putExtra("zipCode", mZipCode);
                startActivity(setAlarmActionIntent);
                return true;

            case R.id.showAboutThisApp:
                Intent showAboutIntent = new Intent(this, AboutThisAppActivity.class);
                startActivity(showAboutIntent);
                return true;

            default:
                return true;
        }
    }

    private void reportErrorToUser() {
        // clear old index number
        resultTextView.setText("Something\nwent\nwrong");
        // clear old index color and info
        UV_indexDescriptionTextView.setText("");
        resultTextView.setBackgroundColor(Color.WHITE);

    }

    private String getTime () {
        Boolean PM = false;

        Calendar c = Calendar.getInstance();
        int hour = c.get(Calendar.HOUR_OF_DAY);

        // Format for AM / PM
        if (hour >= 12) {
            PM = true;
        }
        if (hour > 12) {
            hour -= 12;
        }

        // fix 12 AM
        if (hour == 0) {
            hour = 12;
        }

        // fix no JSON for before 4AM
        if (hour < 4 && !PM) {
            hour = 4;
        }

        // Turn to String
        String hourString = Integer.toString(hour);
        if (hour < 10) {
            hourString = "0" + hourString;
        }

        // formatting
        hourString += " ";
        if (PM) {
            hourString += "PM";
        }
        else {
            hourString += "AM";
        }

        return hourString;
    }

    public void reportLoading () {
        resultTextView.setText("loading");
    }

    private void updateColor (int index, Resources res) {
        // Possible index overflow
        if (index > 11) {
            index = 11;
        }
        String[] colorStringArray = res.getStringArray(R.array.UV_color_code_array);
        String colorString = colorStringArray[index];
        resultTextView.setBackgroundColor(Color.parseColor(colorString));

    }
}


text:
package com.linkedin.pinot.core.segment.index.converter;

import com.linkedin.pinot.core.indexsegment.generator.SegmentVersion;
import org.testng.Assert;
import org.testng.annotations.Test;

public class SegmentFormatConverterFactoryTest {

  @Test
  public void testGetConverter()
      throws Exception {
    {
      SegmentFormatConverter converter = SegmentFormatConverterFactory.getConverter(SegmentVersion.v1, SegmentVersion.v3);
      Assert.assertEquals(converter.getClass(), SegmentV1V2ToV3FormatConverter.class);
    }
    {
      SegmentFormatConverter converter = SegmentFormatConverterFactory.getConverter(SegmentVersion.v2, SegmentVersion.v3);
      Assert.assertEquals(converter.getClass(), SegmentV1V2ToV3FormatConverter.class);
    }
    {
      SegmentFormatConverter converter = SegmentFormatConverterFactory.getConverter(SegmentVersion.v1, SegmentVersion.v2);
      Assert.assertEquals(converter.getClass(), SegmentFormatConverterV1ToV2.class);
    }
  }
}


text:
require "pp"

module Gish
  OK = 0
  ERROR = 1
  NO_MATCHES = 128
  NOT_A_COMMAND = 128
  NOT_A_REPO = 128
  INVALID_OPTION = 129
  TOO_MANY_CHANGES = 49
  NO_CHANGES = 34
end

require_relative "./gish/version"
require_relative "./gish/helpers"
require_relative "./gish/concerns"
require_relative "./gish/commands"
require_relative "./gish/task"
require_relative "./gish/option_parser"


text:
using namespace std;

#define THROW_PQ_ERROR(dbh) throw error_t(PQerrorMessage(dbh))

class PGresultWrap {
protected:
  PGresult* r_;
public:
  PGresultWrap(PGresult* r) : r_(r) {}
  ~PGresultWrap() { PQclear(r_); }
  PGresult* operator*() { return r_; }
private:
  PGresultWrap(const PGresultWrap&); // noncopyable
  PGresultWrap& operator=(const PGresultWrap&);
};

static string escape_quote(const string& src)
{
  string r;
  string::size_type start = 0, quote_at;
  while ((quote_at = src.find('\'', start)) != string::npos) {
    r += src.substr(start, quote_at - start);
    r += "''";
    start = quote_at + 1;
  }
  r += src.substr(start);
  return r;
}

incline_pgsql*
incline_pgsql::factory::create(const string& host, unsigned short port,
			       const string& user, const string& password)
{
  if (port == 0) {
    port = *opt_port_ == 0 ? default_port() : *opt_port_;
  }
  return new incline_pgsql(host.empty() ? *opt_host_ : host, port,
			   user.empty() ? *opt_user_ : user,
			   password.empty() ? *opt_password_ : password);
}

vector<string>
incline_pgsql::factory::create_trigger(const string& name, const string& event,
				       const string& time, const string& table,
				       const map<string, string>& funcvar,
				       const string& funcbody) const
{
  vector<string> r;
  string s("CREATE FUNCTION " + name + "() RETURNS TRIGGER AS '\n");
  if (! funcvar.empty()) {
    s += "DECLARE\n"
      + incline_util::join("",
			   incline_util::filter("  %1 %2;\n", funcvar));
  }
  s += "BEGIN\n" + escape_quote(funcbody) + "  RETURN NEW;\nEND"
    + "' language 'plpgsql'";
  r.push_back(s);
  r.push_back("CREATE TRIGGER " + name + ' ' + time + ' ' + event + " ON "
	      + table + " FOR EACH ROW EXECUTE PROCEDURE " + name + "()");
  return r;
}

vector<string>
incline_pgsql::factory::drop_trigger(const string& name, const string& table,
				     bool if_exists) const
{
  vector<string> r;
  r.push_back(string("DROP TRIGGER ") + (if_exists ? "IF EXISTS " : "") + name
	      + " ON " + table);
  r.push_back(string("DROP FUNCTION ") + (if_exists ? "IF EXISTS " : "") + name
	      + "()");
  return r;
}

string
incline_pgsql::factory::delete_using(const string& table_name,
				     const vector<string>& using_list) const
{
  return "DELETE FROM " + table_name + " USING "
    + incline_util::join(',', using_list);
}

incline_pgsql::~incline_pgsql()
{
  if (dbh_ != NULL) {
    PQfinish(dbh_);
    dbh_ = NULL;
  }
}

string
incline_pgsql::escape(const string& s)
{
  char* buf = (char*)alloca(s.size() * 2 + 1);
  int err = 0;
  PQescapeStringConn(_dbh(), buf, s.c_str(), s.size(), &err);
  if (err != 0) {
    THROW_PQ_ERROR(_dbh());
  }
  return string(buf);
}

unsigned long long
incline_pgsql::execute(const string& stmt)
{
  PGresultWrap ret(PQexec(_dbh(), stmt.c_str()));
  assert(*ret != NULL);
  switch (PQresultStatus(*ret)) {
  case PGRES_COMMAND_OK:
    break;
  case PGRES_EMPTY_QUERY:
  case PGRES_COPY_OUT:
  case PGRES_COPY_IN:
  case PGRES_TUPLES_OK:
    throw error_t("unexpected response from pgsql server");
  default:
    THROW_PQ_ERROR(_dbh());
  }
  unsigned long long nrows = 0;
  if (const char* s = PQcmdTuples(*ret)) {
    if (s[0] != '\0') {
      nrows = strtoll(s, NULL, 10);
    }
  }
  return nrows;
}

void
incline_pgsql::query(vector<vector<value_t> >& rows, const string& stmt)
{
  rows.clear();
  // send query, check response type
  PGresultWrap ret(PQexec(_dbh(), stmt.c_str()));
  assert(*ret != NULL);
  switch (PQresultStatus(*ret)) {
  case PGRES_TUPLES_OK:
    break;
  case PGRES_EMPTY_QUERY:
  case PGRES_COMMAND_OK:
  case PGRES_COPY_OUT:
  case PGRES_COPY_IN:
    throw error_t("unexpected response from pgsql server");
  default:
    THROW_PQ_ERROR(_dbh());
  }
  // read response
  int ntuples = PQntuples(*ret),
    nfields = PQnfields(*ret);
  for (int i = 0; i < ntuples; ++i) {
    rows.push_back(vector<value_t>());
    for (int j = 0; j < nfields; ++j) {
      rows.back().push_back(PQgetvalue(*ret, i, j));
    }
  }
}

string
incline_pgsql::get_column_def(const string& table_name,
			      const string& column_name)
{
  vector<vector<value_t> > rows;
  super::query(rows,
	       "SELECT DATA_TYPE,CHARACTER_MAXIMUM_LENGTH,IS_NULLABLE FROM INFORMATION_SCHEMA.COLUMNS WHERE UDT_CATALOG='%s' AND TABLE_NAME='%s' AND COLUMN_NAME='%s'",
	       escape(*incline_dbms::opt_database_).c_str(),
	       escape(table_name).c_str(),
	       escape(column_name).c_str());
  if (rows.size() != 1) {
    throw error_t("failed to obtain column definition of column:" + table_name
		  + '.' + column_name);
  }
  string def(*rows[0][0]);
  if (! rows[0][1]->empty()) {
    def += '(' + *rows[0][1] + ')';
  }
  if (*rows[0][1] == "NO") {
    def += " NOT NULL";
  }
  return def;
}

incline_pgsql::incline_pgsql(const string& host, unsigned short port,
			     const string& user, const string& password)
  : super(host, port), dbh_(NULL), conninfo_()
{
  stringstream conninfo;
  conninfo << "host=" << host_ << " port=" << port_ << " dbname="
	   << *opt_database_ << " user=" << user << " password=" << password;
  conninfo_ = conninfo.str();
}

PGconn*
incline_pgsql::_dbh()
{
  if (dbh_ == NULL) {
    dbh_ = PQconnectdb(conninfo_.c_str());
    assert(dbh_ != NULL);
    if (PQstatus(dbh_) != CONNECTION_OK) {
      THROW_PQ_ERROR(dbh_);
    }
  }
  return dbh_;
}


text:
using EED.DAL;
using EED.Domain;
using EED.Service.Contests;
using EED.Service.Controller.Account;
using EED.Service.Controller.Contests;
using EED.Service.Controller.Districts;
using EED.Service.Controller.District_Type;
using EED.Service.Controller.Offices;
using EED.Service.Controller.Precincts;
using EED.Service.Controller.Project;
using EED.Service.Controller.User;
using EED.Service.Districts;
using EED.Service.District_Type;
using EED.Service.Election_Type;
using EED.Service.Jurisdiction_Type;
using EED.Service.Membership_Provider;
using EED.Service.Offices;
using EED.Service.Precincts;
using EED.Service.Project;
using Ninject;
using System;
using System.Collections.Generic;
using System.Web.Mvc;
using EED.Service.Political_Party;
using EED.Service.Controller.Political_Party;
using EED.Service.Choices;
using EED.Service.Controller.Choices;

namespace EED.Infrastructure
{
    public class NinjectDependencyResolver : IDependencyResolver
    {
        private readonly IKernel _kernel;

        public NinjectDependencyResolver()
        {
            _kernel = new StandardKernel();
            AddBindings();
        }

        public object GetService(Type serviceType)
        {
            return _kernel.TryGet(serviceType);
        }

        public IEnumerable<object> GetServices(Type serviceType)
        {
            return _kernel.GetAll(serviceType);
        }

        private void AddBindings()
        {
            _kernel.Bind<IRepository<User>>().To<Repository<User>>();
            _kernel.Bind<IRepository<ElectionProject>>().To<Repository<ElectionProject>>();
            _kernel.Bind<IRepository<JurisdictionType>>().To<Repository<JurisdictionType>>();
            _kernel.Bind<IRepository<ElectionType>>().To<Repository<ElectionType>>();
            _kernel.Bind<IRepository<DistrictType>>().To<Repository<DistrictType>>();
            _kernel.Bind<IRepository<District>>().To<Repository<District>>();
            _kernel.Bind<IRepository<Precinct>>().To<Repository<Precinct>>();
            _kernel.Bind<IRepository<Office>>().To<Repository<Office>>();
            _kernel.Bind<IRepository<Contest>>().To<Repository<Contest>>();
            _kernel.Bind<IRepository<PoliticalParty>>().To<Repository<PoliticalParty>>();
            _kernel.Bind<IRepository<Choice>>().To<Repository<Choice>>();

            _kernel.Bind<IMembershipProvider>().To<CustomMembershipProvider>();
            _kernel.Bind<IAuthProvider>().To<FormsAuthProvider>();
            _kernel.Bind<IProjectService>().To<ProjectService>();
            _kernel.Bind<IJurisdictionTypeService>().To<JurisdictionTypeService>();
            _kernel.Bind<IElectionTypeService>().To<ElectionTypeService>();
            _kernel.Bind<IDistrictTypeService>().To<DistrictTypeService>();
            _kernel.Bind<IDistrictService>().To<DistrictService>();
            _kernel.Bind<IPrecinctService>().To<PrecinctService>();
            _kernel.Bind<IOfficeService>().To<OfficeService>();
            _kernel.Bind<IContestService>().To<ContestService>();
            _kernel.Bind<IPoliticalPartyService>().To<PoliticalPartyService>();
            _kernel.Bind<IChoiceService>().To<ChoiceService>();

            _kernel.Bind<IAccountServiceController>().To<AccountServiceController>();
            _kernel.Bind<IUserServiceController>().To<UserServiceController>();
            _kernel.Bind<IProjectServiceController>().To<ProjectServiceController>();
            _kernel.Bind<IDistrictTypeServiceController>().To<DistrictTypeServiceController>();
            _kernel.Bind<IDistrictServiceController>().To<DistrictServiceController>();
            _kernel.Bind<IPrecinctServiceController>().To<PrecinctServiceController>();
            _kernel.Bind<IOfficeServiceController>().To<OfficeServiceController>();
            _kernel.Bind<IContestServiceController>().To<ContestServiceController>();
            _kernel.Bind<IPoliticalPartyServiceController>().To<PoliticalPartyServiceController>();
            _kernel.Bind<IChoiceServiceController>().To<ChoiceServiceController>();
        }
    }
}


text:
var Thermostat = function() {
	this.temperature = 20;
	this.minTemperature = 10;
	this.maxTemperature = 32;
	this.isPowerSaveOn = true;
	this.defaultChange = 1;
	this.PowerSaveMax = 25;
	this.displayColor = 'yellow'
};

Thermostat.prototype.increase = function(passedValue) {
	if (this.isPowerSaveOn) {
		this.increasePowerSaveOn(passedValue);
	} else {
		this.increasePowerSaveOff(passedValue);
	}
	this.assignColor();
};

Thermostat.prototype.increasePowerSaveOn = function(passedValue) {
	if (passedValue == null && (this.temperature+this.defaultChange<this.PowerSaveMax)) {
		this.temperature += this.defaultChange;
	} else if (this.temperature+passedValue<=this.PowerSaveMax) {
		this.temperature += passedValue;
	} else if (this.temperature+passedValue>this.PowerSaveMax) {
		this.temperature = this.PowerSaveMax;
	}
};

Thermostat.prototype.increasePowerSaveOff = function(passedValue) {
	if (passedValue == null && (this.temperature+this.defaultChange<this.maxTemperature)) {
		this.temperature += this.defaultChange;
	} else if (this.temperature+passedValue<=this.maxTemperature) {
		this.temperature += passedValue;
	} else if (this.temperature+passedValue>this.maxTemperature) {
		this.temperature = this.maxTemperature;
	}
};

Thermostat.prototype.decrease = function(passedValue) {
	if (this.isAboveMin() && (passedValue == null)) {
		this.temperature -=1;
	} else if ((this.temperature-passedValue)>=this.minTemperature) {
		this.temperature -= passedValue;
	} else if ((this.temperature-passedValue)<this.minTemperature) {
		this.temperature = this.minTemperature;
	}
	this.assignColor();
};

Thermostat.prototype.isAboveMin = function() {
	return this.temperature > this.minTemperature;
};

Thermostat.prototype.powerSaveSwitch = function() {
	this.isPowerSaveOn = !this.isPowerSaveOn;
};

Thermostat.prototype.reset = function() {
	this.temperature = 20;
};

Thermostat.prototype.assignColor = function() {
	if (this.temperature<18) {
		this.displayColor='green';
	} else if ((this.temperature>=18) && (this.temperature<25)) {
		this.displayColor='yellow';
	} else {
		this.displayColor='red';
	}
};


text:
/*
 *
 *
 *
 *
 *
 * Copyright (c) 2012, Stephen Colebourne & Michael Nascimento Santos
 *
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *
 *  * Redistributions of source code must retain the above copyright notice,
 *    this list of conditions and the following disclaimer.
 *
 *  * Redistributions in binary form must reproduce the above copyright notice,
 *    this list of conditions and the following disclaimer in the documentation
 *    and/or other materials provided with the distribution.
 *
 *  * Neither the name of JSR-310 nor the names of its contributors
 *    may be used to endorse or promote products derived from this software
 *    without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */
package java.time.chrono;

import java.time.DateTimeException;

/**
 * An era in the ISO calendar system.
 *

 * The ISO-8601 standard does not define eras.
 * A definition has therefore been created with two eras - 'Current era' (CE) for
 * years on or after 0001-01-01 (ISO), and 'Before current era' (BCE) for years before that.
 *
 *

 *

 *

 * year-of-era
 * era
 * proleptic-year
 *

 *

 *

 *

 * 2	CE	2
 *

 *

 * 1	CE	1
 *

 *

 * 1	BCE	0
 *

 *

 * 2	BCE	-1
 *

 *

 *

 *

 * Do not use {@code ordinal()} to obtain the numeric representation of {@code IsoEra}.
 * Use {@code getValue()} instead.
 *
 * @implSpec
 * This is an immutable and thread-safe enum.
 *
 * @since 1.8
 */
public enum IsoEra implements Era {

    /**
     * The singleton instance for the era before the current one, 'Before Current Era',
     * which has the numeric value 0.
     */
    BCE,
    /**
     * The singleton instance for the current era, 'Current Era',
     * which has the numeric value 1.
     */
    CE;

    //-----------------------------------------------------------------------
    /**
     * Obtains an instance of {@code IsoEra} from an {@code int} value.
     *

     * {@code IsoEra} is an enum representing the ISO eras of BCE/CE.
     * This factory allows the enum to be obtained from the {@code int} value.
     *
     * @param isoEra  the BCE/CE value to represent, from 0 (BCE) to 1 (CE)
     * @return the era singleton, not null
     * @throws DateTimeException if the value is invalid
     */
    public static IsoEra of(int isoEra) {
        switch (isoEra) {
            case 0:
                return BCE;
            case 1:
                return CE;
            default:
                throw new DateTimeException("Invalid era: " + isoEra);
        }
    }

    //-----------------------------------------------------------------------
    /**
     * Gets the numeric era {@code int} value.
     *

     * The era BCE has the value 0, while the era CE has the value 1.
     *
     * @return the era value, from 0 (BCE) to 1 (CE)
     */
    @Override
    public int getValue() {
        return ordinal();
    }

}


text:
DOCUMENTATION = '''
---
module: php_module
version_added: 2.2
author: "Wong Hoi Sing Edison (@hswong3i)"
short_description: enables/disables a module of the PHP
description:
   - Enables or disables a specified module of the PHP.
options:
   name:
     description:
        - name of the module to enable/disable
     required: true
   state:
     description:
        - indicate the desired state of the resource
     choices: ['present', 'absent']
     default: present
   version:
     description:
        - indicate the version of php for enable/disable
     default: ALL
   sapi:
     description:
        - indicate the sapi of php for enable/disable
     default: ALL

requirements: ["phpenmod","phpdismod"]
'''

EXAMPLES = '''
# enables the PHP module "gd"
- php_module: state=present name=gd

# disables the PHP module "gd"
- php_module: state=absent name=gd

# enables the php module "gd" for php 7.0 only
- php_module: state=present name=gd version=7.0

# enables the php module "gd" for php fpm only
- php_module: state=present name=gd sapi=fpm
'''

RETURN = '''
'''

import re

def _check_module(module):
    name = module.params['name']

    result, stdout, stderr = module.run_command("%s -m %s" % (module.get_bin_path('phpenmod'), name))

    if re.search(r'Not enabling the ' + name + r' module', stderr):
        return 'absent'
    else:
        return 'present'

def _enable_module(module):
    name = module.params['name']
    state = module.params['state']
    version = module.params['version']
    sapi = module.params['sapi']

    current_state = _check_module(module)

    result, stdout, stderr = module.run_command("%s -v %s -s %s %s" % (module.get_bin_path('phpenmod'), version, sapi, name))

    if result != 0:
        module.fail_json(msg = "Failed to enable %s: %s" % (name, stdout))
    else:
        module.exit_json(changed = (current_state != state), result = "%s %s" % (name, state))

def _disable_module(module):
    name = module.params['name']
    state = module.params['state']
    version = module.params['version']
    sapi = module.params['sapi']

    current_state = _check_module(module)

    result, stdout, stderr = module.run_command("%s -v %s -s %s %s" % (module.get_bin_path('phpdismod'), version, sapi, name))

    if result != 0:
        module.fail_json(msg = "Failed to disable %s: %s" % (name, stdout))
    else:
        module.exit_json(changed = (current_state != state), result = "%s %s" % (name, state))

def main():
    module = AnsibleModule(
        argument_spec = dict(
            name  = dict(required = True),
            state = dict(default = 'present', choices = ['absent', 'present']),
            version = dict(required = False, default = 'ALL'),
            sapi = dict(required = False, default = 'ALL'),
        ),
        supports_check_mode = True,
    )

    for binary in ['phpenmod', 'phpdismod']:
        bin_path = module.get_bin_path(binary)
        if bin_path is None:
            module.fail_json(msg = "%s not found" % binary)

    if module.check_mode:
        module.exit_json(changed = (_check_module(module) != module.params['state']))

    if module.params['state'] == 'present':
        _enable_module(module)

    if module.params['state'] == 'absent':
        _disable_module(module)

# import module snippets
from ansible.module_utils.basic import *
if __name__ == '__main__':
    main()