text:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>

    <title>Penlight Documentation</title>
    <link rel="stylesheet" href="../ldoc.css" type="text/css" />

<big></big>

Penlight

Index

Contents

Reading Data Files

Reading Unstructured Text Data

Reading Columnar Data

Reading Configuration Files

Lexical Scanning

XML

Topics

01-introduction.md

02-arrays.md

03-strings.md

04-paths.md

05-dates.md

06-data.md

07-functional.md

08-additional.md

09-discussion.md

Modules

pl

pl.Date

pl.List

pl.Map

pl.MultiMap

pl.OrderedMap

pl.Set

pl.app

pl.array2d

pl.class

pl.comprehension

pl.config

pl.data

pl.dir

pl.file

pl.func

pl.input

pl.lapp

pl.lexer

pl.luabalanced

pl.operator

pl.path

pl.permute

pl.pretty

pl.seq

pl.sip

pl.strict

pl.stringio

pl.stringx

pl.tablex

pl.template

pl.test

pl.text

pl.utils

pl.xml

Examples

seesubst.lua

sipscan.lua

symbols.lua

test-cmp.lua

test-data.lua

test-listcallbacks.lua

test-pretty.lua

test-symbols.lua

testapp.lua

testclone.lua

testconfig.lua

testglobal.lua

testinputfields.lua

testinputfields2.lua

testxml.lua

which.lua

Topic 06-data.md

Data

Reading Data Files

The first thing to consider is this: do you actually need to write a custom file reader? And if the answer is yes, the next question is: can you write the reader in as clear a way as possible? Correctness, Robustness, and Speed; pick the first two and the third can be sorted out later, if necessary.

A common sort of data file is the configuration file format commonly used on Unix systems. This format is often called a property file in the Java world.

 # Read timeout in seconds
 read.timeout=10

 # Write timeout in seconds
 write.timeout=10

Here is a simple Lua implementation:

 -- property file parsing with Lua string patterns
 props = []
 for line in io.lines() do
     if line:find('#,1,true) ~= 1 and not line:find('^%s*$') then
         local var,value = line:match('([^=]+)=(.*)')
         props[var] = value
     end
 end

Very compact, but it suffers from a similar disease in equivalent Perl programs; it uses odd string patterns which are ‘lexically noisy’. Noisy code like this slows the casual reader down. (For an even more direct way of doing this, see the next section, ‘Reading Configuration Files’)

Another implementation, using the Penlight libraries:

 -- property file parsing with extended string functions
 require 'pl'
 stringx.import()
 props = []
 for line in io.lines() do
     if not line:startswith('#') and not line:isspace() then
         local var,value = line:splitv('=')
         props[var] = value
     end
 end

This is more self-documenting; it is generally better to make the code express the intention, rather than having to scatter comments everywhere – comments are necessary, of course, but mostly to give the higher view of your intention that cannot be expressed in code. It is slightly slower, true, but in practice the speed of this script is determined by I/O, so further optimization is unnecessary.

Reading Unstructured Text Data

Text data is sometimes unstructured, for example a file containing words. The pl.input  module has a number of functions which makes processing such files easier. For example, a script to count the number of words in standard input using import.words:

 -- countwords.lua
 require 'pl'
 local k = 1
 for w in input.words(io.stdin) do
     k = k + 1
 end
 print('count',k)

Or this script to calculate the average of a set of numbers using input.numbers :

 -- average.lua
 require 'pl'
 local k = 1
 local sum = 0
 for n in input.numbers(io.stdin) do
     sum = sum + n
     k = k + 1
 end
 print('average',sum/k)

These scripts can be improved further by eliminating loops In the last case, there is a perfectly good function seq.sum  which can already take a sequence of numbers and calculate these numbers for us:

 -- average2.lua
 require 'pl'
 local total,n = seq.sum(input.numbers())
 print('average',total/n)

A further simplification here is that if numbers or words are not passed an argument, they will grab their input from standard input.  The first script can be rewritten:

 -- countwords2.lua
 require 'pl'
 print('count',seq.count(input.words()))

A useful feature of a sequence generator like numbers is that it can read from a string source. Here is a script to calculate the sums of the numbers on each line in a file:

 -- sums.lua
 for line in io.lines() do
     print(seq.sum(input.numbers(line))
 end

Reading Columnar Data

It is very common to find data in columnar form, either space or comma-separated, perhaps with an initial set of column headers. Here is a typical example:

 EventID    Magnitude    LocationX    LocationY    LocationZ
 981124001    2.0    18988.4    10047.1    4149.7
 981125001    0.8    19104.0    9970.4    5088.7
 981127003    0.5    19012.5    9946.9    3831.2
 ...

input.fields  is designed to extract several columns, given some delimiter (default to whitespace).  Here is a script to calculate the average X location of all the events:

 -- avg-x.lua
 require 'pl'
 io.read() -- skip the header line
 local sum,count = seq.sum(input.fields {3})
 print(sum/count)

input.fields  is passed either a field count, or a list of column indices, starting at one as usual. So in this case we're only interested in column 3.  If you pass it a field count, then you get every field up to that count:

 for id,mag,locX,locY,locZ in input.fields (5) do
 ....
 end

input.fields  by default tries to convert each field to a number. It will skip lines which clearly don’t match the pattern, but will abort the script if there are any fields which cannot be converted to numbers.

The second parameter is a delimiter, by default spaces. ‘ ’ is understood to mean ‘any number of spaces’, i.e. ‘%s+’. Any Lua string pattern can be used.

The third parameter is a data source, by default standard input (defined by input.create_getter .) It assumes that the data source has a read method which brings in the next line, i.e. it is a ‘file-like’ object. As a special case, a string will be split into its lines:

 > for x,y in input.fields(2,' ','10 20\n30 40\n') do print(x,y) end
 10      20
 30      40

Note the default behaviour for bad fields, which is to show the offending line number:

 > for x,y in input.fields(2,' ','10 20\n30 40x\n') do print(x,y) end
 10      20
 line 2: cannot convert '40x' to number

This behaviour of input.fields  is appropriate for a script which you want to fail immediately with an appropriate user error message if conversion fails. The fourth optional parameter is an options table: {no_fail=true} means that conversion is attempted but if it fails it just returns the string, rather as AWK would operate. You are then responsible for checking the type of the returned field. {no_convert=true} switches off conversion altogether and all fields are returned as strings.

Sometimes it is useful to bring a whole dataset into memory, for operations such as extracting columns. Penlight provides a flexible reader specifically for reading this kind of data, using the data  module. Given a file looking like this:

 x,y
 10,20
 2,5
 40,50

Then data.read  will create a table like this, with each row represented by a sublist:

 > t = data.read 'test.txt'
 > pretty.dump(t)
 {{10,20},{2,5},{40,50},fieldnames={'x','y'},delim=','}

You can now analyze this returned table using the supplied methods. For instance, the method column_by_name returns a table of all the values of that column.

 -- testdata.lua
 require 'pl'
 d = data.read('fev.txt')
 for _,name in ipairs(d.fieldnames) do
     local col = d:column_by_name(name)
     if type(col[1]) == 'number' then
         local total,n = seq.sum(col)
         utils.printf("Average for %s is %f\n",name,total/n)
     end
 end

data.read  tries to be clever when given data; by default it expects a first line of column names, unless any of them are numbers. It tries to deduce the column delimiter by looking at the first line. Sometimes it guesses wrong; these things can be specified explicitly. The second optional parameter is an options table: can override delim (a string pattern), fieldnames (a list or comma-separated string), specify no_convert (default is to convert), numfields (indices of columns known to be numbers, as a list) and thousands_dot (when the thousands separator in Excel CSV is ‘.’)

A very powerful feature is a way to execute SQL-like queries on such data:

 -- queries on tabular data
 require 'pl'
 local d = data.read('xyz.txt')
 local q = d:select('x,y,z where x > 3 and z < 2 sort by y')
 for x,y,z in q do
     print(x,y,z)
 end

Please note that the format of queries is restricted to the following syntax:

 FIELDLIST [ 'where' CONDITION ] [ 'sort by' FIELD [asc|desc]]

Any valid Lua code can appear in CONDITION; remember it is not SQL and you have to use == (this warning comes from experience.)

For this to work, field names must be Lua identifiers. So read  will massage fieldnames so that all non-alphanumeric chars are replaced with underscores.

read  can handle standard CSV files fine, although doesn’t try to be a full-blown CSV parser. Spreadsheet programs are not always the best tool to process such data, strange as this might seem to some people. This is a toy CSV file; to appreciate the problem, imagine thousands of rows and dozens of columns like this:

 Department Name,Employee ID,Project,Hours Booked
 sales,1231,overhead,4
 sales,1255,overhead,3
 engineering,1501,development,5
 engineering,1501,maintenance,3
 engineering,1433,maintenance,10

The task is to reduce the dataset to a relevant set of rows and columns, perhaps do some processing on row data, and write the result out to a new CSV file. The write_row method uses the delimiter to write the row to a file; Data.select_row is like Data.select, except it iterates over rows, not fields; this is necessary if we are dealing with a lot of columns!

 names = {[1501]='don',[1433]='dilbert'}
 keepcols = {'Employee_ID','Hours_Booked'}
 t:write_row (outf,{'Employee','Hours_Booked'})
 q = t:select_row {
     fields=keepcols,
     where=function(row) return row[1]=='engineering' end
 }
 for row in q do
     row[1] = names[row[1]]
     t:write_row(outf,row)
 end

Data.select_row and Data.select can be passed a table specifying the query; a list of field names, a function defining the condition and an optional parameter sort_by. It isn’t really necessary here, but if we had a more complicated row condition (such as belonging to a specified set) then it is not generally possible to express such a condition as a query string, without resorting to hackery such as global variables.

Data does not have to come from files, nor does it necessarily come from the lab or the accounts department. On Linux, ps aux gives you a full listing of all processes running on your machine. It is straightforward to feed the output of this command into data.read  and perform useful queries on it. Notice that non-identifier characters like ‘%’ get converted into underscores:

 require 'pl'
 f = io.popen 'ps aux'
 s = data.read (f,{last_field_collect=true})
 f:close()
 print(s.fieldnames)
 print(s:column_by_name 'USER')
 qs = 'COMMAND,_MEM where _MEM > 5 and USER=="steve"'
 for name,mem in s:select(qs) do
     print(mem,name)
 end

I've always been an admirer of the AWK programming language; with filter  you can get Lua programs which are just as compact:

 -- printxy.lua
 require 'pl'
 data.filter 'x,y where x > 3'

It is common enough to have data files without headers of field names. data.read  makes a special exception for such files if all fields are numeric. Since there are no column names to use in query expressions, you can use AWK-like column indexes, e.g. ‘$1,$2 where $1 > 3’.  I have a little executable script on my system called lf which looks like this:

 #!/usr/bin/env lua
 require 'pl.data'.filter(arg[1])

And it can be used generally as a filter command to extract columns from data. (The column specifications may be expressions or even constants.)

 $ lf '$1,$5/10' < test.dat

(As with AWK, please note the single-quotes used in this command; this prevents the shell trying to expand the column indexes. If you are on Windows, then you are fine, but it is still necessary to quote the expression in double-quotes so it is passed as one argument to your batch file.)

As a tutorial resource, have a look at test-data.lua  in the PL tests directory for other examples of use, plus comments.

The data returned by read  or constructed by Data.copy_select from a query is basically just an array of rows: {{1,2},{3,4}}. So you may use read  to pull in any array-like dataset, and process with any function that expects such a implementation. In particular, the functions in array2d  will work fine with this data. In fact, these functions are available as methods; e.g. array2d.flatten  can be called directly like so to give us a one-dimensional list:

 v = data.read('dat.txt'):flatten()

The data is also in exactly the right shape to be treated as matrices by LuaMatrix:

 > matrix = require 'matrix'
 > m = matrix(data.read 'mat.txt')
 > = m
 1       0.2     0.3
 0.2     1       0.1
 0.1     0.2     1
 > = m^2  -- same as m*m
 1.07    0.46    0.62
 0.41    1.06    0.26
 0.24    0.42    1.05

write  will write matrices back to files for you.

Finally, for the curious, the global variable _DEBUG can be used to print out the actual iterator function which a query generates and dynamically compiles. By using code generation, we can get pretty much optimal performance out of arbitrary queries.

 > lua -lpl -e "_DEBUG=true" -e "data.filter 'x,y where x > 4 sort by x'" < test.txt
 return function (t)
         local i = 0
         local v
         local ls = {}
         for i,v in ipairs(t) do
             if v[1] > 4  then
                     ls[#ls+1] = v
             end
         end
         table.sort(ls,function(v1,v2)
             return v1[1] < v2[1]
         end)
         local n = #ls
         return function()
             i = i + 1
             v = ls[i]
             if i > n then return end
             return v[1],v[2]
         end
 end

 10,20
 40,50

Reading Configuration Files

The config  module provides a simple way to convert several kinds of configuration files into a Lua table. Consider the simple example:

 # test.config
 # Read timeout in seconds
 read.timeout=10

 # Write timeout in seconds
 write.timeout=5

 #acceptable ports
 ports = 1002,1003,1004

This can be easily brought in using config.read  and the result shown using pretty.write :

 -- readconfig.lua
 local config = require 'pl.config'
 local pretty= require 'pl.pretty'

 local t = config.read(arg[1])
 print(pretty.write(t))

and the output of lua readconfig.lua test.config is:

 {
   ports = {
     1002,
     1003,
     1004
   },
   write_timeout = 5,
   read_timeout = 10
 }

That is, config.read  will bring in all key/value pairs, ignore # comments, and ensure that the key names are proper Lua identifiers by replacing non-identifier characters with ‘_’. If the values are numbers, then they will be converted. (So the value of t.write_timeout is the number 5). In addition, any values which are separated by commas will be converted likewise into an array.

Any line can be continued with a backslash. So this will all be considered one line:

 names=one,two,three, \
 four,five,six,seven, \
 eight,nine,ten

Windows-style INI files are also supported. The section structure of INI files translates naturally to nested tables in Lua:

 ; test.ini
 [timeouts]
 read=10 ; Read timeout in seconds
 write=5 ; Write timeout in seconds
 [portinfo]
 ports = 1002,1003,1004

 The output is:

 {
   portinfo = {
     ports = {
       1002,
       1003,
       1004
     }
   },
   timeouts = {
     write = 5,
     read = 10
   }
 }

You can now refer to the write timeout as t.timeouts.write.

As a final example of the flexibility of config.read , if passed this simple comma-delimited file

 one,two,three
 10,20,30
 40,50,60
 1,2,3

it will produce the following table:

 {
   { "one", "two", "three" },
   { 10, 20, 30 },
   { 40, 50, 60  },
   { 1, 2, 3 }
 }

config.read  isn’t designed to read all CSV files in general, but intended to support some Unix configuration files not structured as key-value pairs, such as ‘/etc/passwd’.

This function is intended to be a Swiss Army Knife of configuration readers, but it does have to make assumptions, and you may not like them. So there is an optional extra parameter which allows some control, which is table that may have the following fields:

 {
    variablilize = true,
    convert_numbers = true,
    trim_space = true,
    list_delim = ','
 }

variablilize is the option that converted write.timeout in the first example to the valid Lua identifier write_timeout.  If convert_numbers is true, then an attempt is made to convert any string that starts like a number. trim_space ensures that there is no starting or trailing whitespace with values, and list_delim is the character that will be used to decide whether to split a value up into a list (it may be a Lua string pattern such as ‘%s+’.)

For instance, the password file in Unix is colon-delimited:

 t = config.read('/etc/passwd',{list_delim=':'})

This produces the following output on my system (only last two lines shown):

 {
   ...
   {
     "user",
     "x",
     "1000",
     "1000",
     "user,,,",
     "/home/user",
     "/bin/bash"
   },
   {
     "sdonovan",
     "x",
     "1001",
     "1001",
     "steve donovan,28,,",
     "/home/sdonovan",
     "/bin/bash"
   }
 }

You can get this into a more sensible format, where the usernames are the keys, with:

 t = tablex.pairmap(function(k,v) return v,v[1] end,t)

and you get:

 { ...
   sdonovan = {
     "sdonovan",
     "x",
     "1001",
     "1001",
     "steve donovan,28,,",
     "/home/sdonovan",
     "/bin/bash"
   }
 ...
 }

Lexical Scanning

Although Lua’s string pattern matching is very powerful, there are times when something more powerful is needed.  pl.lexer.scan  provides lexical scanners which tokenizes a string, classifying tokens into numbers, strings, etc.

 > lua -lpl
 Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
 > tok = lexer.scan 'alpha = sin(1.5)'
 > = tok()
 iden    alpha
 > = tok()
 =       =
 > = tok()
 iden    sin
 > = tok()
 (       (
 > = tok()
 number  1.5
 > = tok()
 )       )
 > = tok()
 (nil)

The scanner is a function, which is repeatedly called and returns the type and value of the token.  Recognized basic types are ‘iden’,‘string’,‘number’, and ‘space’. and everything else is represented by itself. Note that by default the scanner will skip any ‘space’ tokens.

‘comment’ and ‘keyword’ aren’t applicable to the plain scanner, which is not language-specific, but a scanner which understands Lua is available. It recognizes the Lua keywords, and understands both short and long comments and strings.

 > for t,v in lexer.lua 'for i=1,n do' do print(t,v) end
 keyword for
 iden    i
 =       =
 number  1
 ,       ,
 iden    n
 keyword do

A lexical scanner is useful where you have highly-structured data which is not nicely delimited by newlines. For example, here is a snippet of a in-house file format which it was my task to maintain:

 points    (818344.1,-20389.7,-0.1),(818337.9,-20389.3,-0.1),(818332.5,-20387.8,-0.1)
     ,(818327.4,-20388,-0.1),(818322,-20387.7,-0.1),(818316.3,-20388.6,-0.1)
     ,(818309.7,-20389.4,-0.1),(818303.5,-20390.6,-0.1),(818295.8,-20388.3,-0.1)
     ,(818290.5,-20386.9,-0.1),(818285.2,-20386.1,-0.1),(818279.3,-20383.6,-0.1)
     ,(818274,-20381.2,-0.1),(818274,-20380.7,-0.1);

Here is code to extract the points using pl.lexer :

 -- assume 's' contains the text above...
 local lexer = require 'pl.lexer'
 local expecting = lexer.expecting
 local append = table.insert

 local tok = lexer.scan(s)

 local points = {}
 local t,v = tok() -- should be 'iden','points'

 while t ~= ';' do
     c = {}
     expecting(tok,'(')
     c.x = expecting(tok,'number')
     expecting(tok,',')
     c.y = expecting(tok,'number')
     expecting(tok,',')
     c.z = expecting(tok,'number')
     expecting(tok,')')
     t,v = tok()  -- either ',' or ';'
     append(points,c)
 end

The expecting function grabs the next token and if the type doesn’t match, it throws an error. (pl.lexer , unlike other PL libraries, raises errors if something goes wrong, so you should wrap your code in pcall  to catch the error gracefully.)

The scanners all have a second optional argument, which is a table which controls whether you want to exclude spaces and/or comments. The default for lexer.lua  is {space=true,comments=true}.  There is a third optional argument which determines how string and number tokens are to be processsed.

The ultimate highly-structured data is of course, program source. Here is a snippet from ‘text-lexer.lua’:

 require 'pl'

 lines = [[
 for k,v in pairs(t) do
     if type(k) == 'number' then
         print(v) -- array-like case
     else
         print(k,v)
     end
 end
 ]]

 ls = List()
 for tp,val in lexer.lua(lines,{space=true,comments=true}) do
     assert(tp ~= 'space' and tp ~= 'comment')
     if tp == 'keyword' then ls:append(val) end
 end
 test.asserteq(ls,List{'for','in','do','if','then','else','end','end'})

Here is a useful little utility that identifies all common global variables found in a lua module:

 -- testglobal.lua
 require 'pl'

 local txt,err = utils.readfile(arg[1])
 if not txt then return print(err) end

 local globals = List()
 for t,v in lexer.lua(txt) do
     if t == 'iden' and _G[v] then
         globals:append(v)
     end
 end
 pretty.dump(seq.count_map(globals))

Rather then dumping the whole list, with its duplicates, we pass it through seq.count_map  which turns the list into a table where the keys are the values, and the associated values are the number of times those values occur in the sequence. Typical output looks like this:

 {
   type = 2,
   pairs = 2,
   table = 2,
   print = 3,
   tostring = 2,
   require = 1,
   ipairs = 4
 }

You could further pass this through tablex.keys  to get a unique list of symbols. This can be useful when writing ‘strict’ Lua modules, where all global symbols must be defined as locals at the top of the file.

For a more detailed use of lexer.scan , please look at testxml.lua  in the examples directory.

XML

New in the 0.9.7 release is some support for XML. This is a large topic, and Penlight does not provide a full XML stack, which is properly the task of a more specialized library.

Parsing and Pretty-Printing

The semi-standard XML parser in the Lua universe is lua-expat. In particular, it has a function called lxp.lom.parse which will parse XML into the Lua Object Model (LOM) format. However, it does not provide a way to convert this data back into XML text.  xml.parse  will use this function, if lua-expat is available, and otherwise switches back to a pure Lua parser originally written by Roberto Ierusalimschy.

The resulting document object knows how to render itself as a string, which is useful for debugging:

 > d = xml.parse "<nodes><node id='1'>alice</node></nodes>"
 > = d
 <nodes><node id='1'>alice</node></nodes>
 > pretty.dump (d)
 {
   {
     "alice",
     attr = {
       "id",
       id = "1"
     },
     tag = "node"
   },
   attr = {
   },
   tag = "nodes"
 }

Looking at the actual shape of the data reveals the structure of LOM:

every element has a tag field with its name

plus a attr field which is a table containing the attributes as fields, and also as an array. It is always present.

the children of the element are the array part of the element, so d[1] is the first child of d, etc.

It could be argued that having attributes also as the array part of attr is not essential (you generally cannot depend on attribute order in XML) but that’s how it goes with this standard.

lua-expat is another soft dependency of Penlight; generally, the fallback parser is good enough for straightforward XML as is commonly found in configuration files, etc. doc.basic_parse is not intended to be a proper conforming parser (it’s only sixty lines) but it handles simple kinds of documents that do not have comments or DTD directives. It is intelligent enough to ignore the <?xml directive and that is about it.

You can get pretty-printing by explicitly calling xml.tostring  and passing it the initial indent and the per-element indent:

 > = xml.tostring(d,'','  ')

 <nodes>
   <node id='1'>alice</node>
 </nodes>

There is a fourth argument which is the attribute indent:

 > a = xml.parse "<frodo name='baggins' age='50' type='hobbit'/>"
 > = xml.tostring(a,'','  ','  ')

 <frodo
   type='hobbit'
   name='baggins'
   age='50'
 />

Parsing and Working with Configuration Files

It’s common to find configurations expressed with XML these days. It’s straightforward to ‘walk’ the LOM data and extract the data in the form you want:

 require 'pl'

 local config = [[
 <config>
     <alpha>1.3</alpha>
     <beta>10</beta>
     <name>bozo</name>
 </config>
 ]]
 local d,err = xml.parse(config)

 local t = {}
 for item in d:childtags() do
     t[item.tag] = item[1]
 end

 pretty.dump(t)
 --->
 {
   beta = "10",
   alpha = "1.3",
   name = "bozo"
 }

The only gotcha is that here we must use the Doc:childtags method, which will skip over any text elements.

A more involved example is this excerpt from serviceproviders.xml, which is usually found at /usr/share/mobile-broadband-provider-info/serviceproviders.xml on Debian/Ubuntu Linux systems.

 d = xml.parse [[
 <serviceproviders format="2.0">
 <country code="za">
     <provider>
         <name>Cell-c</name>
         <gsm>
             <network-id mcc="655" mnc="07"/>
             <apn value="internet">
                 <username>Cellcis</username>
                 <dns>196.7.0.138</dns>
                 <dns>196.7.142.132</dns>
             </apn>
         </gsm>
     </provider>
     <provider>
         <name>MTN</name>
         <gsm>
             <network-id mcc="655" mnc="10"/>
             <apn value="internet">
                 <dns>196.11.240.241</dns>
                 <dns>209.212.97.1</dns>
             </apn>
         </gsm>
     </provider>
     <provider>
         <name>Vodacom</name>
         <gsm>
             <network-id mcc="655" mnc="01"/>
             <apn value="internet">
                 <dns>196.207.40.165</dns>
                 <dns>196.43.46.190</dns>
             </apn>
             <apn value="unrestricted">
                 <name>Unrestricted</name>
                 <dns>196.207.32.69</dns>
                 <dns>196.43.45.190</dns>
             </apn>
         </gsm>
     </provider>
     <provider>
         <name>Virgin Mobile</name>
         <gsm>
             <apn value="vdata">
                 <dns>196.7.0.138</dns>
                 <dns>196.7.142.132</dns>
             </apn>
         </gsm>
     </provider>
 </country>

 </serviceproviders>
 ]]

Getting the names of the providers per-country is straightforward:

 local t = {}
 for country in d:childtags() do
     local providers = {}
     t[country.tag] = providers
     for provider in country:childtags() do
         table.insert(providers,provider:child_with_name('name'):get_text())
     end
 end

 pretty.dump(t)
 -->
 {
   country = {
     "Cell-c",
     "MTN",
     "Vodacom",
     "Virgin Mobile"
   }
 }

Generating XML with ‘xmlification’

This feature is inspired by the htmlify function used by Orbit to simplify HTML generation, except that no function environment magic is used; the tags function returns a set of constructors for elements of the given tag names.

 > nodes, node = xml.tags 'nodes, node'
 > = node 'alice'
 <node>alice</node>
 > = nodes { node {id='1','alice'}}
 <nodes><node id='1'>alice</node></nodes>

The flexibility of Lua tables is very useful here, since both the attributes and the children of an element can be encoded naturally. The argument to these tag constructors is either a single value (like a string) or a table where the attributes are the named keys and the children are the array values.

Generating XML using Templates

A template is a little XML document which contains dollar-variables. The subst method on a document is fed an array of tables containing values for these variables. Note how the parent tag name is specified:

 > templ = xml.parse "<node id='$id'>$name</node>"
 > = templ:subst {tag='nodes', {id=1,name='alice'},{id=2,name='john'}}
 <nodes><node id='1'>alice</node><node id='2'>john</node></nodes>

Extracting Data using Templates

Matching goes in the opposite direction.  We have a document, and would like to extract values from it using a pattern.

A common use of this is parsing the XML result of API queries.  The (undocumented) Google Weather API is a good example. Grabbing the result of http://www.google.com/ig/api?weather=Johannesburg,ZA" we get something like this, after pretty-printing:

 <xml_api_reply version='1'>
   <weather module_id='0' tab_id='0' mobile_zipped='1' section='0' row='0' mobile_row='0'>
     <forecast_information>
       <city data='Johannesburg, Gauteng'/>
       <postal_code data='Johannesburg,ZA'/>
       <latitude_e6 data=''/>
       <longitude_e6 data=''/>
       <forecast_date data='2010-10-02'/>
       <current_date_time data='2010-10-02 18:30:00 +0000'/>
       <unit_system data='US'/>
     </forecast_information>
     <current_conditions>
       <condition data='Clear'/>
       <temp_f data='75'/>
       <temp_c data='24'/>
       <humidity data='Humidity: 19%'/>
       <icon data='/ig/images/weather/sunny.gif'/>
       <wind_condition data='Wind: NW at 7 mph'/>
     </current_conditions>
     <forecast_conditions>
       <day_of_week data='Sat'/>
       <low data='60'/>
       <high data='89'/>
       <icon data='/ig/images/weather/sunny.gif'/>
       <condition data='Clear'/>
     </forecast_conditions>
     ....
    </weather>
 </xml_api_reply>

Assume that the above XML has been read into google. The idea is to write a pattern looking like a template, and use it to extract some values of interest:

 t = [[
   <weather>
     <current_conditions>
       <condition data='$condition'/>
       <temp_c data='$temp'/>
     </current_conditions>
   </weather>
 ]]

 local res, ret = google:match(t)
 pretty.dump(res)

And the output is:

 {
   condition = "Clear",
   temp = "24"
 }

The match method can be passed a LOM document or some text, which will be parsed first. Note that $NUMBER is treated specially as a numerical index, so that $1 is the first element of the resulting array, etc.

generated by LDoc 1.2


text:
var args = process.argv.slice(2);
var TIMESTAMP = ((args[args.indexOf("--timestamp")] === ("--timestamp")) ? args[args.indexOf("--timestamp")+1] : "");
var GUARDIAN = ((args[args.indexOf("--guardian")] === ("--guardian")) ? args[args.indexOf("--guardian")+1] : "");
var dateString = (new Date(parseInt(TIMESTAMP))).toISOString().substr(0,19).replace(/:/g,"-");
var s3Path = "/"+dateString.substr(0,4)+"/"+dateString.substr(5,2)+"/"+dateString.substr(8,2)+"/"+GUARDIAN+"/"+GUARDIAN+"-"+dateString;
console.log(s3Path+'*'+(new Date(parseInt(TIMESTAMP))).toISOString()+'*');

redpajama_set_name: RedPajamaGithub


text:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

<meta http-equiv="Content-Type" content="text/html" charset="UTF-8">
<title>Uses of Interface opennlp.tools.parser.GapLabeler (Apache OpenNLP Tools 1.6.0 API)</title>
<meta name="date" content="2015-06-16">
<link rel="stylesheet" type="text/css" href="../../../../stylesheet.css" title="Style">

JavaScript is disabled on your browser.

Overview

Package

Class

Use

Tree

Deprecated

Index

Help

Prev

Next

Frames

No Frames

All Classes

Uses of Interface
opennlp.tools.parser.GapLabeler

Packages that use GapLabeler

Package
Description

opennlp.tools.parser

Package containing common code for performing full syntactic parsing.

opennlp.tools.parser.lang.en

opennlp.tools.parser.lang.es

Uses of GapLabeler in opennlp.tools.parser

Methods in opennlp.tools.parser with parameters of type GapLabeler

Modifier and Type
Method and Description

static Parse
Parse.parseParse(String parse,
          GapLabeler gl)

Parses the specified tree-bank style parse string and return a Parse structure
 for that string.

Uses of GapLabeler in opennlp.tools.parser.lang.en

Classes in opennlp.tools.parser.lang.en that implement GapLabeler

Modifier and Type
Class and Description

class
HeadRules

Class for storing the English head rules associated with parsing.

Uses of GapLabeler in opennlp.tools.parser.lang.es

Classes in opennlp.tools.parser.lang.es that implement GapLabeler

Modifier and Type
Class and Description

class
AncoraSpanishHeadRules

Class for storing the Ancora Spanish head rules associated with parsing.

Overview

Package

Class

Use

Tree

Deprecated

Index

Help

Prev

Next

Frames

No Frames

All Classes

Copyright © 2015 The Apache Software Foundation. All rights reserved.


text:
#ifndef RDESTL_HASH_MAP_H
#define RDESTL_HASH_MAP_H

#include "algorithm.h"
#include "allocator.h"
#include "functional.h"
#include "hash.h"
#include "pair.h"

namespace rde
{

// TLoadFactor4 - controls hash map load. 4 means 100% load, ie. hashmap will grow
// when number of items == capacity. Default value of 6 means it grows when
// number of items == capacity * 3/2 (6/4). Higher load == tighter maps, but bigger
// risk of collisions.
template<typename TKey, typename TValue,
		class THashFunc = rde::hash<TKey>,
		int TLoadFactor4 = 6,
		class TKeyEqualFunc = rde::equal_to<TKey>,
		class TAllocator = rde::allocator>
class hash_map
{
public:
	typedef rde::pair<TKey, TValue>         value_type;

private:
	struct node
    {
		static const hash_value_t kUnusedHash       = 0xFFFFFFFF;
        static const hash_value_t kDeletedHash      = 0xFFFFFFFE;

        node(): hash(kUnusedHash) {}

        RDE_FORCEINLINE bool is_unused() const		{ return hash == kUnusedHash; }
        RDE_FORCEINLINE bool is_deleted() const     { return hash == kDeletedHash; }
        RDE_FORCEINLINE bool is_occupied() const	{ return hash < kDeletedHash; }

		hash_value_t    hash;
        value_type      data;
	};
	template<typename TNodePtr, typename TPtr, typename TRef>
    class node_iterator
    {
		friend class hash_map;
    public:
		typedef forward_iterator_tag    iterator_category;

        explicit node_iterator(TNodePtr node, const hash_map* map)
        :	m_node(node),
            m_map(map)
		{/**/}
        template<typename UNodePtr, typename UPtr, typename URef>
        node_iterator(const node_iterator<UNodePtr, UPtr, URef>& rhs)
        :	m_node(rhs.node()),
	  		m_map(rhs.get_map())
		{/**/}

		TRef operator*() const
        {
			RDE_ASSERT(m_node != 0);
            return m_node->data;
		}
        TPtr operator->() const
        {
			return &m_node->data;
		}
        RDE_FORCEINLINE TNodePtr node() const
        {
			return m_node;
		}

		node_iterator& operator++()
        {
			RDE_ASSERT(m_node != 0);
			++m_node;
            move_to_next_occupied_node();
			return *this;
        }
		node_iterator operator++(int)
        {
			node_iterator copy(*this);
            ++(*this);
            return copy;
		}

        RDE_FORCEINLINE bool operator==(const node_iterator& rhs) const
        {
			return rhs.m_node == m_node;
		}
        bool operator!=(const node_iterator& rhs) const
        {
			return !(rhs == *this);
		}

        const hash_map* get_map() const { return m_map; }
	private:
		void move_to_next_occupied_node()
        {
			// @todo: save nodeEnd in constructor?
			TNodePtr nodeEnd = m_map->m_nodes + m_map->bucket_count();
            for (/**/; m_node < nodeEnd; ++m_node)
            {
				if (m_node->is_occupied())
					break;
			}
		}
        TNodePtr		m_node;
        const hash_map*	m_map;
	};

public:
	typedef TKey																key_type;
    typedef TValue																mapped_type;
    typedef TAllocator															allocator_type;
	typedef node_iterator<node*, value_type*, value_type&>						iterator;
	typedef node_iterator<const node*, const value_type*, const value_type&>	const_iterator;
    typedef int																	size_type;
	static const size_type														kNodeSize = sizeof(node);
	static const size_type														kInitialCapacity = 64;

	hash_map()
	:	m_nodes(&ms_emptyNode),
		m_size(0),
		m_capacity(0),
		m_capacityMask(0),
		m_numUsed(0)
	{
		RDE_ASSERT((kInitialCapacity & (kInitialCapacity - 1)) == 0);	// Must be power-of-two
	}
	explicit hash_map(const allocator_type& allocator)
    :	m_nodes(&ms_emptyNode),
        m_size(0),
        m_capacity(0),
		m_capacityMask(0),
        m_numUsed(0),
        m_allocator(allocator)
	{
		/**/
	}
	explicit hash_map(size_type initial_bucket_count,
		const allocator_type& allocator = allocator_type())
    :	m_nodes(&ms_emptyNode),
        m_size(0),
        m_capacity(0),
		m_capacityMask(0),
        m_numUsed(0),
        m_allocator(allocator)
	{
		reserve(initial_bucket_count);
	}
	hash_map(size_type initial_bucket_count,
		const THashFunc& hashFunc,
		const allocator_type& allocator = allocator_type())
    :	m_nodes(&ms_emptyNode),
        m_size(0),
        m_capacity(0),
		m_capacityMask(0),
        m_numUsed(0),
		m_hashFunc(hashFunc),
        m_allocator(allocator)
	{
		reserve(initial_bucket_count);
	}
	hash_map(const hash_map& rhs, const allocator_type& allocator = allocator_type())
    :	m_nodes(&ms_emptyNode),
        m_size(0),
        m_capacity(0),
		m_capacityMask(0),
        m_numUsed(0),
        m_allocator(allocator)
	{
		*this = rhs;
	}
	explicit hash_map(e_noinitialize)
	{
		/**/
	}
	~hash_map()
	{
		delete_nodes();
	}

	iterator begin()
    {
		iterator it(m_nodes, this);
		it.move_to_next_occupied_node();
		return it;
	}
	const_iterator begin() const
    {
		const_iterator it(m_nodes, this);
		it.move_to_next_occupied_node();
		return it;
	}
	iterator end()              { return iterator(m_nodes + m_capacity, this); }
	const_iterator end() const	{ return const_iterator(m_nodes + m_capacity, this); }

	// @note:	Added for compatiblity sake.
	//			Personally, I consider it "risky". Use find/insert for more
	//			explicit operations.
	mapped_type& operator[](const key_type& key)
	{
		hash_value_t hash;
		node* n = find_for_insert(key, &hash);
		if (n == 0 || !n->is_occupied())
		{
			return insert_at(value_type(key, TValue()), n, hash).first->second;
		}
		return n->data.second;
	}
	// @note:	Doesn't copy allocator.
	hash_map& operator=(const hash_map& rhs)
	{
		RDE_ASSERT(invariant());
		if (&rhs != this)
		{
			clear();
			if (m_capacity < rhs.bucket_count())
			{
				delete_nodes();
				m_nodes = allocate_nodes(rhs.bucket_count());
				m_capacity = rhs.bucket_count();
				m_capacityMask = m_capacity - 1;
			}
			rehash(m_capacity, m_nodes, rhs.m_capacity, rhs.m_nodes, false);
			m_size = rhs.size();
			m_numUsed = rhs.m_numUsed;
		}
		RDE_ASSERT(invariant());
		return *this;
	}
	void swap(hash_map& rhs)
	{
		if (&rhs != this)
		{
			RDE_ASSERT(invariant());
			RDE_ASSERT(m_allocator == rhs.m_allocator);
			rde::swap(m_nodes, rhs.m_nodes);
			rde::swap(m_size, rhs.m_size);
			rde::swap(m_capacity, rhs.m_capacity);
			rde::swap(m_capacityMask, rhs.m_capacityMask);
			rde::swap(m_numUsed, rhs.m_numUsed);
			rde::swap(m_hashFunc, rhs.m_hashFunc);
			rde::swap(m_keyEqualFunc, rhs.m_keyEqualFunc);
			RDE_ASSERT(invariant());
		}
	}

	rde::pair<iterator, bool> insert(const value_type& v)
	{
		typedef rde::pair<iterator, bool> ret_type_t;
		RDE_ASSERT(invariant());
		if (m_numUsed * TLoadFactor4 >= m_capacity * 4)
			grow();

		hash_value_t hash;
		node* n = find_for_insert(v.first, &hash);
		if (n->is_occupied())
		{
			RDE_ASSERT(hash == n->hash && m_keyEqualFunc(v.first, n->data.first));
			return ret_type_t(iterator(n, this), false);
		}
		if (n->is_unused())
			++m_numUsed;
		rde::copy_construct(&n->data, v);
        n->hash = hash;
        ++m_size;
		RDE_ASSERT(invariant());
        return ret_type_t(iterator(n, this), true);
	}

	size_type erase(const key_type& key)
    {
		node* n = lookup(key);
        if (n != (m_nodes + m_capacity) && n->is_occupied())
        {
			erase_node(n);
            return 1;
		}
		return 0;
	}
    void erase(iterator it)
    {
		RDE_ASSERT(it.get_map() == this);
        if (it != end())
        {
			RDE_ASSERT(!empty());
            erase_node(it.node());
		}
	}
	void erase(iterator from, iterator to)
	{
		for (/**/; from != to; ++from)
        {
			node* n = from.node();
            if (n->is_occupied())
				erase_node(n);
		}
	}

	iterator find(const key_type& key)
    {
		node* n = lookup(key);
        return iterator(n, this);
	}
	const_iterator find(const key_type& key) const
	{
		const node* n = lookup(key);
		return const_iterator(n, this);
	}

	void clear()
    {
		node* endNode = m_nodes + m_capacity;
        for (node* iter = m_nodes; iter != endNode; ++iter)
        {
            if( iter )
            {
                if (iter->is_occupied())
                {
                    rde::destruct(&iter->data);
                }
                // We can make them unused, because we clear whole hash_map,
                // so we can guarantee there'll be no holes.
                iter->hash = node::kUnusedHash;
            }
		}
        m_size = 0;
        m_numUsed = 0;
	}

	void reserve(size_type min_size)
	{
		size_type newCapacity = (m_capacity == 0 ? kInitialCapacity : m_capacity);
		while (newCapacity < min_size)
			newCapacity *= 2;
		if (newCapacity > m_capacity)
			grow(newCapacity);
	}

	size_type bucket_count() const			{ return m_capacity; }
	size_type size() const					{ return m_size; }
	size_type empty() const					{ return size() == 0; }
	size_type nonempty_bucket_count() const	{ return m_numUsed; }
	size_type used_memory() const
	{
		return bucket_count() * kNodeSize;
	}

	const allocator_type& get_allocator() const	{ return m_allocator; }
	void set_allocator(const allocator_type& allocator)
	{
		m_allocator = allocator;
	}

private:
	void grow()
	{
		const int newCapacity = (m_capacity == 0 ? kInitialCapacity : m_capacity * 2);
		grow(newCapacity);
	}
	void grow(int new_capacity)
	{
		RDE_ASSERT((new_capacity & (new_capacity - 1)) == 0);	// Must be power-of-two
		node* newNodes = allocate_nodes(new_capacity);
		rehash(new_capacity, newNodes, m_capacity, m_nodes, true);
		if (m_nodes != &ms_emptyNode)
			m_allocator.deallocate(m_nodes, sizeof(node) * m_capacity);
		m_capacity = new_capacity;
		m_capacityMask = new_capacity - 1;
		m_nodes = newNodes;
		m_numUsed = m_size;
		RDE_ASSERT(m_numUsed < m_capacity);
   }
	rde::pair<iterator, bool> insert_at(const value_type& v, node* n,
		hash_value_t hash)
	{
		RDE_ASSERT(invariant());
		if (n == 0 || m_numUsed * TLoadFactor4 >= m_capacity * 4)
			return insert(v);

		RDE_ASSERT(!n->is_occupied());
		if (n->is_unused())
			++m_numUsed;
		rde::copy_construct(&n->data, v);
		n->hash = hash;
		++m_size;
		RDE_ASSERT(invariant());
		return rde::pair<iterator, bool>(iterator(n, this), true);
	}
	node* find_for_insert(const key_type& key, hash_value_t* out_hash)
	{
		if (m_capacity == 0)
			return 0;

		const hash_value_t hash = hash_func(key);
        *out_hash = hash;
        uint32 i = hash & m_capacityMask;

        node* n = m_nodes + i;
		if (n->hash == hash && m_keyEqualFunc(key, n->data.first))
			return n;

        node* freeNode(0);
        if (n->is_deleted())
			freeNode = n;
		uint32 numProbes(0);
        // Guarantees loop termination.
        RDE_ASSERT(m_numUsed < m_capacity);
        while (!n->is_unused())
        {
			++numProbes;
            i = (i + numProbes) & m_capacityMask;
            n = m_nodes + i;
			if (compare_key(n, key, hash))
				return n;
            if (n->is_deleted() && freeNode == 0)
				freeNode = n;
		}
        return freeNode ? freeNode : n;
	}
	node* lookup(const key_type& key) const
	{
		const hash_value_t hash = hash_func(key);
        uint32 i = hash & m_capacityMask;
        node* n = m_nodes + i;
		if (n->hash == hash && m_keyEqualFunc(key, n->data.first))
			return n;

		uint32 numProbes(0);
        // Guarantees loop termination.
        RDE_ASSERT(m_capacity == 0 || m_numUsed < m_capacity);
        while (!n->is_unused())
        {
			++numProbes;
            i = (i + numProbes) & m_capacityMask;
            n = m_nodes + i;

			if (compare_key(n, key, hash))
				return n;
		}
		return m_nodes + m_capacity;
	}

	static void rehash(int new_capacity, node* new_nodes,
		int capacity, const node* nodes, bool destruct_original)
	{
        //if (nodes == &ms_emptyNode || new_nodes == &ms_emptyNode)
          //  return;

		const node* it = nodes;
		const node* itEnd = nodes + capacity;
		const uint32 mask = new_capacity - 1;
		while (it != itEnd)
		{
			if (it->is_occupied())
			{
				const hash_value_t hash = it->hash;
				uint32 i = hash & mask;

				node* n = new_nodes + i;
				uint32 numProbes(0);
				while (!n->is_unused())
				{
					++numProbes;
					i = (i + numProbes) & mask;
					n = new_nodes + i;
				}
				rde::copy_construct(&n->data, it->data);
				n->hash = hash;
				if (destruct_original)
					rde::destruct(&it->data);
			}
			++it;
		}
	}

	node* allocate_nodes(int n)
	{
		node* buckets = static_cast<node*>(m_allocator.allocate(n * sizeof(node)));
        node* iterBuckets(buckets);
        node* end = iterBuckets + n;
        for (/**/; iterBuckets != end; ++iterBuckets)
			iterBuckets->hash = node::kUnusedHash;

		return buckets;
	}
	void delete_nodes()
	{
		node* it = m_nodes;
        node* itEnd = it + m_capacity;
        while (it != itEnd)
        {
			if (it && it->is_occupied())
				rde::destruct(&it->data);
			++it;
		}
		if (m_nodes != &ms_emptyNode)
			m_allocator.deallocate(m_nodes, sizeof(node) * m_capacity);

        m_capacity = 0;
		m_capacityMask = 0;
        m_size = 0;
	}
	void erase_node(node* n)
	{
		RDE_ASSERT(!empty());
        RDE_ASSERT(n->is_occupied());
        rde::destruct(&n->data);
        n->hash = node::kDeletedHash;
		--m_size;
	}

	RDE_FORCEINLINE hash_value_t hash_func(const key_type& key) const
	{
		const hash_value_t h = m_hashFunc(key) & 0xFFFFFFFD;
		//RDE_ASSERT(h < node::kDeletedHash);
        return h;
	}
	bool invariant() const
	{
		RDE_ASSERT((m_capacity & (m_capacity - 1)) == 0);
		RDE_ASSERT(m_numUsed >= m_size);
		return true;
	}

	RDE_FORCEINLINE bool compare_key(const node* n, const key_type& key, hash_value_t hash) const
	{
		return (n->hash == hash && m_keyEqualFunc(key, n->data.first));
	}

	node*			m_nodes;
	int				m_size;
	int				m_capacity;
	uint32			m_capacityMask;
	int				m_numUsed;
	THashFunc       m_hashFunc;
	TKeyEqualFunc	m_keyEqualFunc;
	TAllocator      m_allocator;

	static node		ms_emptyNode;
};

// Holy ...
template<typename TKey, typename TValue,
		class THashFunc,
		int TLoadFactor4,
		class TKeyEqualFunc,
		class TAllocator>
typename hash_map<TKey, TValue, THashFunc, TLoadFactor4, TKeyEqualFunc, TAllocator>::node hash_map<TKey, TValue, THashFunc, TLoadFactor4, TKeyEqualFunc, TAllocator>::ms_emptyNode;

}

#endif


text:
class AddFlyerFileToEpisodes < ActiveRecord::Migration[5.0]
  def change
    add_column :episodes, :flyer_file_id, :string
  end
end


text:
sudo chown vagrant:vagrant /opt/

### Update the system
sudo apt-get update

# echo mysql-server mysql-server/root_password password password | sudo debconf-set-selections
# echo mysql-server mysql-server/root_password_again password password | sudo debconf-set-selections

### Install system dependencies
sudo apt-get install -y build-essential curl gcc g++ git libaio1 libaio-dev nfs-common openssl

### NodeJS ###

### Node 4.4.5
# Download the binary
# wget http://nodejs.org/dist/v4.4.5/node-v4.4.5-linux-x64.tar.gz -O /tmp/node-v4.4.5-linux-x64.tar.gz

# Unpack it
# cd /tmp
# tar -zxvf /tmp/node-v4.4.5-linux-x64.tar.gz
# mv /tmp/node-v4.4.5-linux-x64 /opt/node-v4.4.5-linux-x64
# ln -s /opt/node-v4.4.5-linux-x64 /opt/nodejs

# MY NODE INSTALL SHIT

curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get install -y nodejs

# Set the node_path
export NODE_PATH=/opt/nodejs/lib/node_modules
export NODE_PATH=$NODE_PATH:/opt/dev/node_modules
export NODE_PATH=$NODE_PATH:/opt/dev/lib/node_modules
export NODE_PATH=$NODE_PATH:/usr/local/lib/node_modules

# Install global Node dependencies
# /opt/nodejs/bin/npm install -g n
sudo npm install -g n

# /opt/nodejs/bin/npm config set loglevel http
sudo npm config set loglevel http

### Add binaries to path ###

# First run the command
# export PATH=$PATH:/opt/mongodb/bin:/opt/mysql/server-5.6/bin:/opt/nodejs/bin
export NODE_PATH=/opt/nodejs/lib/node_modules
export NODE_PATH=$NODE_PATH:/opt/dev/node_modules
export NODE_PATH=$NODE_PATH:/opt/dev/lib/node_modules
export NODE_PATH=$NODE_PATH:/usr/local/lib/node_modules/lib/node_modules:/usr/local/lib/node_modules

# Now save to the /etc/bash.bashrc file so it works on reboot
cp /etc/bash.bashrc /tmp/bash.bashrc
printf "\n#Add binaries to path\n\nexport PATH=$PATH:/opt/mongodb/bin:/opt/mysql/server-5.6/bin:/opt/nodejs/bin\nexport NODE_PATH=/opt/nodejs/lib/node_modules\nexport NODE_PATH=$NODE_PATH:/opt/dev/node_modules\nexport NODE_PATH=$NODE_PATH:/opt/dev/lib/node_modules" > /tmp/path
cat /tmp/path >> /tmp/bash.bashrc
sudo chown root:root /tmp/bash.bashrc
sudo mv /tmp/bash.bashrc /etc/bash.bashrc

### Update the /etc/hosts file ###
printf '127.0.0.1       localhost\n127.0.1.1       debian-squeeze.caris.de debian-squeeze nodebox\n\n# The following lines are desirable for IPv6 capable hosts\n::1     ip6-localhost ip6-loopback\nfe00::0 ip6-localnet\nff00::0 ip6-mcastprefix\nff02::1 ip6-allnodes\nff02::2 ip6-allrouters' > /tmp/hosts
sudo mv /tmp/hosts /etc/hosts

### Install Git Aware Prompt ###
mkdir ~/.bash
cd ~/.bash
git clone git://github.com/jimeh/git-aware-prompt.git

printf 'export GITAWAREPROMPT=~/.bash/git-aware-prompt\nsource $GITAWAREPROMPT/main.sh\n\nexport PS1="\${debian_chroot:+(\\$debian_chroot)}\\[\\033[01;32m\\]\\u@\\h\\[\\033[00m\\]:\\[\\033[01;34m\\]\\w\\[\\033[00m\\] \\[$txtcyn\\]\\$git_branch\\[$txtred\\]\\$git_dirty\\[$txtrst\\]\$ "' > ~/.bash_profile

### Set a message of the day ###
sudo rm /etc/motd
sudo cp /vagrant/vagrant-scripts/motd.txt /etc/motd

### Test that everything is installed ok ###
printf "\n\n--- Running post-install checks ---\n\n"
node /vagrant/vagrant-scripts/postInstall.js

### Finished ###
printf "\n\n--- NodeBox is now built ---\n\n"

### MY STUFF
# sudo npm install -g npm@latest --no-bin-links
sudo npm install -g node-pre-gyp --no-bin-links
# npm install -g @angular/cli --no-bin-links
sudo npm install http-server -g
sudo npm install sqlite3@3.1.8 -g

printf "\n\n--- App system prerequisites are now installed. Running npm install... ---\n\n"

sudo mkdir /home/vagrant/node_modules
sudo ln -s /home/vagrant/node_modules/ /opt/dev/backend/node_modules

sudo rm -rf /opt/dev/backend/node_modules/*
cd /opt/dev/backend/ && sudo npm install --no-bin-links

printf "\n\n--- NPM install for the server completed ---\n Run 'vagrant ssh' to access the VM \n"

# node /opt/dev/backend/index.js &

export PATH=$PATH:~/bin
sudo cp /vagrant/vagrant-scripts/runall.sh /bin
sudo mv /bin/runall.sh /bin/runall

# /vagrant/runall.sh

# cd /opt/dev/backend/ && node index.js &


text:
package org.elbek.chord.core;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Socket;
import java.util.concurrent.BlockingDeque;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.LinkedBlockingDeque;

/**
 * this class is wrapper for socket class that provides thread safe access with call back function thus enabling reuse of the same socket while making usage thread safe
 * Created by elbek on 9/22/17.
 */
public class SocketRunner implements Runnable {
    private ReferenceNode referenceNode;
    private Node node;
    private int maxRetry = 2;
    private Socket socket;
    private volatile boolean stop = false;
    private BlockingDeque<SocketTask> tasks = new LinkedBlockingDeque<>();

    //this creates new socket, thus creating new thread on the server side, create this object with care
    public SocketRunner(ReferenceNode referenceNode, Node node) throws IOException {
        this.socket = referenceNode.newSocket();
        this.referenceNode = referenceNode;
        this.node = node;
    }

    @Override
    public void run() {
        int currentTry = 0;
        boolean releaseLatch = true;
        while (true) {
            SocketTask socketTask = null;
            try {
                if (tasks.isEmpty() && Thread.currentThread().isInterrupted()) {
                    break;
                }
                socketTask = tasks.take();
                socketTask.doIt(socket, referenceNode);
                currentTry = 0;
                releaseLatch = true;
            } catch (InterruptedException e) {
                e.printStackTrace();
                stop = true;
            } catch (IOException e) {
                if (currentTry != maxRetry) {
                    try {
                        recover();
                        currentTry++;
                        tasks.add(socketTask);
                        releaseLatch = false;
                    } catch (IOException e1) {
                        e1.printStackTrace();
                        stop = true;
                    }
                } else {
                    stop = true;
                }
                e.printStackTrace();
            } finally {
                if (releaseLatch && socketTask != null && socketTask.latch != null) {
                    socketTask.latch.countDown();
                }
            }
        }
        try {
            socket.close();
        } catch (IOException e) {
        }
    }

    private void recover() throws IOException {
        try {
            socket.close();
        } catch (IOException ignored) {
        }
        socket = new Socket();
        socket.connect(new InetSocketAddress(referenceNode.host, referenceNode.port));
    }

    /**
     * add task to task queue
     * returns true if it was able to add, false if socket is already stopped
     * To add properly use {@link SocketRunner#add(Resolver, SocketTask)} method
     *
     * @param task
     * @return
     */
    public boolean add(SocketTask task) {
        if (stop) {
            return false;
        }
        tasks.add(task);
        return true;
    }

    @Override
    public String toString() {
        return "SocketRunner{" + referenceNode + '}';
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        SocketRunner that = (SocketRunner) o;

        return referenceNode.equals(that.referenceNode);
    }

    @Override
    public int hashCode() {
        return referenceNode.hashCode();
    }

    public synchronized void stop() {
        stop = true;
        Thread.currentThread().interrupt(); //in case queue is blocked waiting new socket commands
    }

    public Socket getSocket() {
        return socket;
    }

    public static abstract class SocketTask {
        CountDownLatch latch;

        public SocketTask() {
        }

        public SocketTask(CountDownLatch latch) {
            this.latch = latch;
        }

        abstract void doIt(Socket socket, ReferenceNode referenceNode) throws IOException;
    }

    public interface Resolver {
        ReferenceNode resolve();
    }

    /**
     * this method tries to add socket task to socket task runner object of reference node
     * it repeats until it succeeds
     *
     * @param resolver,  the functor class to resolve Reference node
     * @param socketTask task that runs in SocketRunner
     */
    public static void add(Resolver resolver, SocketTask socketTask, Node node) throws IOException {
        SocketRunner socketRunner;
        do {
            ReferenceNode resolvedNode = resolver.resolve();
            if (resolvedNode == null) {
                continue;
            }
            if (resolvedNode.isSelfNode(node)) { //looks like no socket needed
                try {
                    socketTask.doIt(null, resolvedNode);
                } finally {
                    if (socketTask.latch != null) {
                        socketTask.latch.countDown();
                    }
                }
                break;
            }
            socketRunner = node.getSocketManager().getSocketRunner(resolvedNode);
            if (socketRunner != null && socketRunner.add(socketTask)) {
                return;
            }
        } while (true);
    }
}


text:
# -*- coding: utf-8 -*-
require 'rubygems'
require 'vortex_client'
require 'nokogiri'
require 'open-uri'
require 'uri'
require 'pry'
require "net/http"
require 'pathname'
require 'json'
require 'pp'

# Scrape content from the Norwegian Center for Studies of Holocaust and Religious Minorities'
# website http://www.hlsenteret.no/ and re-publish content to University of Oslo's CMS
# using the WebDAV API.

# TODO
#  ok - Hente ut tittel på mappen
#  ok - Hente ut content/type på bilder
#  ok - Kopiere over ingress bilde
#  ok - Luke ut rare tegn i overskriftene
#  ok - Kjøre alle dokumenter uten kræsj
#  ok - Kopiere over flere bilder
#  ok - Få logging til fil til å fungere
# - Logg publisering til fil
# - Hvorfor konverteres og publiseres /konv/kunnskapsbasen/-a-hrefhttp-.html
# - Håndtere /konv/kunnskapsbasen/hl-senterets-kunnskapsbase.html spesielt?

@vortex = Vortex::Connection.new("https://nyweb4-dav.uio.no", :osx_keychain => true)

# Simple logger
def log(str)
  puts str
  File.open("scrape_holocaust.log", 'a') do |f|
    f.write( Time.now.iso8601 + ";" + str + "\n" )
  end
end

def http_content_type(url)
  uri = URI.parse(url)
  http = Net::HTTP.new(uri.host, uri.port)

  request = Net::HTTP::Get.new(uri.request_uri)
  request["User-Agent"] = "My Ruby Script"
  request["Accept"] = "*/*"

  response = http.request(request)
  return response['content-type']
end

def scrape_folder_title(url)
  begin
    doc = Nokogiri::HTML.parse(open(scrape_url))
    doc.encoding = 'utf-8'

    title = nil
    begin
      title = doc.css(".folder .title").first.inner_html
    rescue
    end
    if(title == nil)then
      title = doc.css(".article .title").first.inner_html
    end
  rescue
  end
  return title
end

def create_path(dest_path)
  destination_path = "/"
  dest_path.split("/").each do |folder|
    if(folder != "")then
      folder = folder.downcase
      destination_path = destination_path + folder + "/"

      if( not(@vortex.exists?(destination_path)) )then

        puts "Creating folder " + destination_path

        uri = URI.parse(@url)

        title = scrape_folder_title( 'http://' + uri.host + destination_path.gsub('/konv','') )

        if(title)then
          title = title[0..0].upcase + title[1..title.length]
          puts "Mappetittel: " + title.to_s
        end

        @vortex.mkdir(destination_path)
        @vortex.proppatch(destination_path,'<v:collection-type xmlns:v="vrtx">article-listing</v:collection-type>')
        if(title)then
          @vortex.proppatch(destination_path,'<v:userTitle xmlns:v="vrtx">' + title.to_s +  '</v:userTitle>')
        end
      end
    end
  end
end

# Return an array images and captions
def scrape_images(doc)
  images = []
  i = 0
  doc.css(".imageSeriesImage").each do |item|
    url = item.attr("src")
    caption = doc.css(".imageText")[i].text
    images << { :url => url, :caption => caption }
    i = i + 1
  end
  return images
end

# Resize an image using the unix command line utility 'sips' available on osx
def resize_image(content, content_type, size)
  filename = "/tmp/" + (1 + rand(10000000)).to_s + "." + content_type
  filename_resized = "/tmp/" + (1 + rand(10000000)).to_s + "_resized." + content_type
  File.open(filename, 'w') do |f|
    f.write( content)
  end
  result = %x[sips --resampleWidth #{size} #{filename} --out #{filename_resized}]
  content_resized = IO.readlines(filename_resized,'r').to_s
  return content_resized
end

# Download graphic to dest_path and return absolute filename
def download_image(src_url,dest_path)
  src_url = src_url.gsub(/\?.*/,'')
  content_type = http_content_type(src_url)
  content_type = content_type.gsub("image/", "").gsub("jpeg","jpg")
  content = open(src_url).read
  basename = Pathname.new(src_url).basename.to_s.gsub(/\..*/,'')
  vortex_url = dest_path + basename + "." + content_type
  vortex_url = vortex_url.downcase
  begin
    @vortex.put_string(vortex_url, content)
  rescue Exception => e
    puts e.message
    pp e.backtrace.inspect
    puts "vortex_url: " + vortex_url
    exit
  end

  # Store a resized image to vortex
  puts "Nedskalerer bilde: " + src_url
  content_resized = resize_image(content, content_type,300)
  vortex_url_resized = dest_path + basename + "_width_300." + content_type

  vortex_url_resized = vortex_url_resized.downcase
  begin
    @vortex.put_string(vortex_url_resized, content_resized)
  rescue Exception => e
    puts e.message
    pp e.backtrace.inspect
    puts "vortex_url_resized: " + vortex_url_resized
    exit
  end

  return { :vortex_url_resized => vortex_url_resized, :vortex_url => vortex_url}
end

def publish_article(path, title, introduction, body, doc)
  dest_path = "/konv" + path.gsub(/\d*$/,'')

  if(not(dest_path =~ /\/$/))then
    dest_path = dest_path + "/"
  end

  puts "Publishing to: '#{dest_path}'"

  create_path(dest_path)
  @vortex.cd(dest_path)

  # images = []
  # Comment out this line to prevent scraper from downloading images
  images = scrape_images(doc)

  images.each do |image|
    filenames = download_image(image[:url], dest_path)
    image[:vortex_url] = filenames[:vortex_url_resized]
    image[:vortex_url_org] = filenames[:vortex_url]
  end

  url = dest_path + Vortex::StringUtils.create_filename(title.gsub(":","_")).gsub(/-$/,'') + '.html'
  url = url.downcase

  attributes = {:title => title,
    :introduction => introduction,
    :body => body,
    :publishedDate => Time.now,
    :url => url}

  if(images and images.first)then
    attributes[:picture] = images.first[:vortex_url]
  end

  article  = Vortex::StructuredArticle.new(attributes )
  begin
    url = @vortex.publish(article)
  rescue Exception => e
    puts e.message
    require 'pp'
    pp e.backtrace.inspect
    pp attributes
    puts "Path: " + dest_path
    exit
  end

  # Add additional images to bottom of page
  images_html = ""
  if(images and images.size > 1)then
    images[1..images.size].each do |image|
      image_html = <<EOF

                #{image[:caption]}

EOF
      images_html = images_html + image_html
    end
  end

  # Reopen document and set caption on article image
  if(images and images.first)then
    @vortex.find(url) do |item|
      data = JSON.parse(item.content)

      caption = ""
      if(images.first[:caption])then
        caption = images.first[:caption]
      end
      caption = caption + " Last ned i full størrelse"
      data["properties"]["caption"] = caption

      # Add addiitional images at bottom
      data["properties"]["content"] = data["properties"]["content"] + images_html
      item.content = data.to_json
    end
  end

  return url
end

def scrape_article(url)
  doc = Nokogiri::HTML.parse(open(url))
  doc.encoding = 'utf-8'
  if(doc.css(".article .title").size() == 0)then
    puts "Warning. No title. Ignoring: " + url
    return
  end
  title = doc.css(".article .title").first.inner_html
  introduction = ""
  begin
    introduction = doc.css(".article .abstract").first.inner_html
  rescue
  end

  body = ""
  doc.css(".article .text").each do |p|
    body = body + "
" + p.inner_html + "
"
  end

  title = title.gsub('–','–') # Fjern bindestrek

  uri = URI.parse(url)
  path = uri.path

  # Remove inline css
  introduction = introduction.gsub(/style=\"[^\"]*\"/,"")
  body = body.gsub(/style=\"[^\"]*\"/,"")

  published_path = publish_article(path, title, introduction, body, doc)
  # log = path + ";" + URI.parse(published_path).path
  log = url + ";" + published_path + "\n"
  File.open("scrape_holocaust.log", 'a') {|f| f.write(log) }
  puts title + " => " + published_path
end

def http_status_code(url)
  uri = URI.parse(url)
  Net::HTTP.start(uri.host, uri.port) do |http|
    return http.head(uri.request_uri).code.to_i
  end
end

def scrape_article_listing(url)

  if( http_status_code(url) == 404)then
    puts "Advarsel: Status code 404: " + url
    return
  end

  # Ad-hoc rule to ignore path
  if(url =~ /\/kunnskapsbasen\/Presse/)then
    puts "Advarsel: Ignorerer pressesidene har for store bilder for vortex: url"
    return
  end

  doc = Nokogiri::HTML.parse(open(url))
  doc.encoding = 'utf-8'

  if(doc.css(".folder .list").size > 0 )then
    puts "Scraping article listing page: " + url
    doc.css(".folder .list .article").each do |article|
      href = article.css(".title a").attr("href").text
      begin
        href = href.gsub("%20","")
      rescue
      end
      scrape_article_listing(href)
    end
  else
    # Pages without .folder .list is articles
    puts "Scraping article: " + url
    scrape_article(url)
  end

end

# if @vortex.exists?('/konv/kunnskapsbasen/aktor') then
#  @vortex.delete('/konv/kunnskapsbasen/aktor')
# end
@url = "http://www.hlsenteret.no/kunnskapsbasen/"
File.open("scrape_holocaust.log", 'w') {|f| f.write("") }
scrape_article_listing(@url)

# Denne har 4 bilder
# scrape_article("http://www.hlsenteret.no/kunnskapsbasen/folkemord/armenerne/1334")

# Denne har ingen tittel og skal ignoreres
# scrape_article('http://www.hlsenteret.no/kunnskapsbasen/tradisjoner/86')

# Denne returnerer 404
# scrape_article_listing("http://www.hlsenteret.no/kunnskapsbasen/tema/kunst")

# Denne har bilder som er for store vortex
# scrape_article_listing("http://www.hlsenteret.no/kunnskapsbasen/Presse/")

# scrape_article('http://www.hlsenteret.no/kunnskapsbasen/tema/relpol/10129')


text:
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System.Reflection;
using System.Security.Cryptography.Asn1;
using Test.Cryptography;
using Xunit;

namespace System.Security.Cryptography.Tests.Asn1
{
    internal static class AsnReaderExtensions
    {
        private delegate Asn1Tag RTAL(out int? parsedLength, out int bytesRead);

        public static Asn1Tag ReadTagAndLength(this AsnReader reader, out int? parsedLength, out int bytesRead)
        {
            return ((RTAL)
                typeof(AsnReader).GetMethod("ReadTagAndLength", BindingFlags.Instance | BindingFlags.NonPublic)
                    .CreateDelegate(typeof(RTAL), reader)).Invoke(out parsedLength, out bytesRead);
        }
    }

    public sealed class ReadLength : Asn1ReaderTests
    {
        [Theory]
        [InlineData(4, 0, "0400")]
        [InlineData(1, 1, "0101")]
        [InlineData(4, 127, "047F")]
        [InlineData(4, 128, "048180")]
        [InlineData(4, 255, "0481FF")]
        [InlineData(2, 256, "02820100")]
        [InlineData(4, int.MaxValue, "04847FFFFFFF")]
        public static void MinimalPrimitiveLength(int tagValue, int length, string inputHex)
        {
            byte[] inputBytes = inputHex.HexToByteArray();

            foreach (PublicEncodingRules rules in Enum.GetValues(typeof(PublicEncodingRules)))
            {
                AsnReader reader = new AsnReader(inputBytes, (AsnEncodingRules)rules);

                Asn1Tag tag = reader.ReadTagAndLength(out int ? parsedLength, out int bytesRead);

                Assert.Equal(inputBytes.Length, bytesRead);
                Assert.False(tag.IsConstructed, "tag.IsConstructed");
                Assert.Equal(tagValue, tag.TagValue);
                Assert.Equal(length, parsedLength.Value);

                // ReadTagAndLength doesn't move the _data span forward.
                Assert.True(reader.HasData, "reader.HasData");
            }
        }

        [Theory]
        [InlineData(-1)]
        [InlineData(3)]
        public static void ReadWithUnknownRuleSet(int invalidRuleSetValue)
        {
            byte[] data = { 0x05, 0x00 };

            Assert.Throws<ArgumentOutOfRangeException>(
                () => new AsnReader(data, (AsnEncodingRules)invalidRuleSetValue));
        }

        [Theory]
        [InlineData("")]
        [InlineData("05")]
        [InlineData("0481")]
        [InlineData("048201")]
        [InlineData("04830102")]
        [InlineData("0484010203")]
        public static void ReadWithInsufficientData(string inputHex)
        {
            byte[] inputData = inputHex.HexToByteArray();
            AsnReader reader = new AsnReader(inputData, AsnEncodingRules.DER);

            Assert.Throws<CryptographicException>(() => reader.ReadTagAndLength(out _, out _));
        }

        [Theory]
        [InlineData("DER indefinite constructed", PublicEncodingRules.DER, "3080" + "0500" + "0000")]
        [InlineData("0xFF-BER", PublicEncodingRules.BER, "04FF")]
        [InlineData("0xFF-CER", PublicEncodingRules.CER, "04FF")]
        [InlineData("0xFF-DER", PublicEncodingRules.DER, "04FF")]
        [InlineData("CER definite constructed", PublicEncodingRules.CER, "30820500")]
        [InlineData("BER indefinite primitive", PublicEncodingRules.BER, "0480" + "0000")]
        [InlineData("CER indefinite primitive", PublicEncodingRules.CER, "0480" + "0000")]
        [InlineData("DER indefinite primitive", PublicEncodingRules.DER, "0480" + "0000")]
        [InlineData("DER non-minimal 0", PublicEncodingRules.DER, "048100")]
        [InlineData("DER non-minimal 7F", PublicEncodingRules.DER, "04817F")]
        [InlineData("DER non-minimal 80", PublicEncodingRules.DER, "04820080")]
        [InlineData("CER non-minimal 0", PublicEncodingRules.CER, "048100")]
        [InlineData("CER non-minimal 7F", PublicEncodingRules.CER, "04817F")]
        [InlineData("CER non-minimal 80", PublicEncodingRules.CER, "04820080")]
        [InlineData("BER too large", PublicEncodingRules.BER, "048480000000")]
        [InlineData("CER too large", PublicEncodingRules.CER, "048480000000")]
        [InlineData("DER too large", PublicEncodingRules.DER, "048480000000")]
        [InlineData("BER padded too large", PublicEncodingRules.BER, "0486000080000000")]
        [InlineData("BER uint.MaxValue", PublicEncodingRules.BER, "0484FFFFFFFF")]
        [InlineData("CER uint.MaxValue", PublicEncodingRules.CER, "0484FFFFFFFF")]
        [InlineData("DER uint.MaxValue", PublicEncodingRules.DER, "0484FFFFFFFF")]
        [InlineData("BER padded uint.MaxValue", PublicEncodingRules.BER, "048800000000FFFFFFFF")]
        [InlineData("BER 5 byte spread", PublicEncodingRules.BER, "04850100000000")]
        [InlineData("CER 5 byte spread", PublicEncodingRules.CER, "04850100000000")]
        [InlineData("DER 5 byte spread", PublicEncodingRules.DER, "04850100000000")]
        [InlineData("BER padded 5 byte spread", PublicEncodingRules.BER, "0486000100000000")]
        public static void InvalidLengths(
            string description,
            PublicEncodingRules rules,
            string inputHex)
        {
            byte[] inputData = inputHex.HexToByteArray();
            AsnReader reader = new AsnReader(inputData, (AsnEncodingRules)rules);

            Assert.Throws<CryptographicException>(() => reader.ReadTagAndLength(out _, out _));
        }

        [Theory]
        [InlineData(PublicEncodingRules.BER)]
        [InlineData(PublicEncodingRules.CER)]
        public static void IndefiniteLength(PublicEncodingRules ruleSet)
        {
            // SEQUENCE (indefinite)
            //   NULL
            //   End-of-Contents
            byte[] data = { 0x30, 0x80, 0x05, 0x00, 0x00, 0x00 };
            AsnReader reader = new AsnReader(data, (AsnEncodingRules)ruleSet);

            Asn1Tag tag = reader.ReadTagAndLength(out int? length, out int bytesRead);

            Assert.Equal(2, bytesRead);
            Assert.False(length.HasValue, "length.HasValue");
            Assert.Equal((int)UniversalTagNumber.Sequence, tag.TagValue);
            Assert.True(tag.IsConstructed, "tag.IsConstructed");
        }

        [Theory]
        [InlineData(0, "0483000000")]
        [InlineData(1, "048A00000000000000000001")]
        [InlineData(128, "049000000000000000000000000000000080")]
        public static void BerNonMinimalLength(int expectedLength, string inputHex)
        {
            byte[] inputData = inputHex.HexToByteArray();
            AsnReader reader = new AsnReader(inputData, AsnEncodingRules.BER);

            Asn1Tag tag = reader.ReadTagAndLength(out int? length, out int bytesRead);

            Assert.Equal(inputData.Length, bytesRead);
            Assert.Equal(expectedLength, length.Value);
            // ReadTagAndLength doesn't move the _data span forward.
            Assert.True(reader.HasData, "reader.HasData");
        }

        [Theory]
        [InlineData(PublicEncodingRules.BER, 4, 0, 5, "0483000000" + "0500")]
        [InlineData(PublicEncodingRules.DER, 1, 1, 2, "0101" + "FF")]
        [InlineData(PublicEncodingRules.CER, 0x10, null, 2, "3080" + "0500" + "0000")]
        public static void ReadWithDataRemaining(
            PublicEncodingRules ruleSet,
            int tagValue,
            int? expectedLength,
            int expectedBytesRead,
            string inputHex)
        {
            byte[] inputData = inputHex.HexToByteArray();
            AsnReader reader = new AsnReader(inputData, (AsnEncodingRules)ruleSet);

            Asn1Tag tag = reader.ReadTagAndLength(out int? length, out int bytesRead);

            Assert.Equal(expectedBytesRead, bytesRead);
            Assert.Equal(tagValue, tag.TagValue);
            Assert.Equal(expectedLength, length);
        }
    }
}


text:
require 'uri'
require 'json'
require 'trollop'
require 'faraday'
require 'faraday_middleware'

class RestApi
  VERSION = "2.3.0-pre".freeze
  API_CMD = File.basename($PROGRAM_NAME)

  class Cli
    METHODS = {
      "get"     => "get",
      "put"     => "put",
      "post"    => "post",
      "patch"   => "patch",
      "edit"    => "post",
      "create"  => "post",
      "update"  => "post",
      "delete"  => "delete",
      "options" => "options",
    }.freeze

    SEP       = ("_" * 60).freeze
    PREFIX    = "/api".freeze
    CTYPE     = "application/json".freeze
    SCRIPTDIR = ENV['API_SCRIPTDIR'] ? ENV['API_SCRIPTDIR'] : ENV['HOME'] + "/bin".freeze

    ACTIONS              = METHODS.keys
    METHODS_NEEDING_DATA = %w(put post patch).freeze
    SCRIPTDIR_ACTIONS    = %w(ls run).freeze
    SUB_COMMANDS         = ACTIONS + %w(edit vi) + SCRIPTDIR_ACTIONS
    API_PARAMETERS       = %w(expand attributes decorators limit offset
                              depth search_options
                              sort_by sort_order sort_options
                              filter by_tag provider_class requester_type).freeze

    MULTI_PARAMS         = %w(filter).freeze

    def msg_exit(msg, exit_code = 1)
      puts msg
      exit exit_code
    end

    def json_pretty(json)
      puts JSON.pretty_generate(JSON.parse(json))
    rescue
      puts json
    end

    def prompt_get_data
      data = ""
      puts "Enter data to send with request:"
      puts "Terminate with \"\" or \".\""
      while (s = gets)
        s = s.strip
        break if s == "." || s == ""
        data << s
      end
      data
    end

    def run
      data      = ""
      path      = ""
      params    = {}

      opts = Trollop.options do
        version "#{API_CMD} #{VERSION} - ManageIQ REST API Access Script"
        banner <<-EOS
      #{version}

      Usage: #{API_CMD} [options] <action> [parameters] [resource]

                  action - is the action to use for the request, i.e. get, post, patch, edit ...

                  [parameters] include: #{API_PARAMETERS.join(", ")}
                               specify --help for additional help

                  [resource] - is the optional resource i.e. services

             #{API_CMD} [options] vi|edit [script]

                  Edit optional api_* scripts. script names must be specified without the
                  api_ prefix or .rb suffix. Edits this script if not specified.

             #{API_CMD} [options] run script [method]

                  Run optional api_* scripts

             #{API_CMD} [options] ls

                  List optional api_* scripts (without the api_ prefix)

      #{API_CMD} options are:
      EOS
        opt :verbose,    "Verbose mode, show details of the communication",
            :default => false,                    :short => '-v'
        opt :apiversion, "Version of the API to access",
            :default => "",                       :short => '-V'
        opt :url,        "Base URL of Appliance to access",
            :default => "http://localhost:3000",  :short => '-l'
        opt :user,       "User to authenticate as",
            :default => "admin",                  :short => '-u'
        opt :group,      "User group to authorize as",
            :default => "",                       :short => '-g'
        opt :password,   "Password for user specified to authenticate as",
            :default => "smartvm",                :short => '-p'
        opt :token,      "Token to use for authentication instead of user/password",
            :default => "",                       :short => '-t'
        opt :miqtoken,   "Token to use for system authentication",
            :default => "",                       :short => '-m'
        opt :format,     "How to format Json, pretty|none",
            :default => "pretty",                 :short => '-f'
        opt :inputfile,  "File to use as input to the POST/PUT/PATCH methods",
            :default => "",                       :short => '-i'
        opt :scriptdir,  "Directory where optional api_* scripts live",
            :default => SCRIPTDIR,                :short => '-s'
        stop_on SUB_COMMANDS
      end

      unless opts[:inputfile].empty?
        Trollop.die :inputfile, "File specified #{opts[:inputfile]} does not exist" unless File.exist?(opts[:inputfile])
      end

      begin
        URI.parse(opts[:url])
      rescue
        Trollop.die :url, "Invalid URL syntax specified #{opts[:url]}"
      end

      action = ARGV.shift
      Trollop.die "Must specify an action" if action.nil?

      if SCRIPTDIR_ACTIONS.include?(action)
        msg_exit("Script directory #{opts[:scriptdir]} does not exist") unless File.directory?(opts[:scriptdir])
      end

      if action == "ls"
        d = Dir.open(opts[:scriptdir])
        d.each do |file|
          p = file.scan(/^api_(.*)\.rb/)
          puts p unless p.nil?
        end
        d.close
        exit 0
      end

      if action == "vi" || action == "edit"
        api_script = ARGV.shift
        api_script_file = if api_script.nil? || api_script == ""
                            File.expand_path($PROGRAM_NAME)
                          else
                            File.join(opts[:scriptdir], "api_#{api_script}.rb")
                          end
        ed_cmd = "vi"
        ed_cmd = ENV["EDITOR"] if action == "edit" && ENV["EDITOR"]
        cmd = "#{ed_cmd} #{api_script_file}"
        system(cmd)
        exit 0
      end

      if action == "run"
        script = ARGV.shift
        method = ARGV.shift
        msg_exit("Must specify a script to run.") if script.nil?
        api_script = "#{opts[:scriptdir]}/api_#{script}.rb"
        msg_exit("Script file #{api_script} does not exist") unless File.exist?(api_script)
      else
        api_params = Trollop.options do
          norm_options  = {:default => ""}
          multi_options = {:default => "", :multi => true}
          API_PARAMETERS.each { |p| opt p.intern, p, (MULTI_PARAMS.include?(p) ? multi_options.dup : norm_options.dup) }
        end
        API_PARAMETERS.each { |param| params[param] = api_params[param.intern] unless api_params[param.intern].empty? }
      end

      if action != "run"
        resource = ARGV.shift

        resource = "/" + resource             if resource && resource[0] != "/"
        resource = resource.gsub(PREFIX, '')  unless resource.nil?

        method = METHODS[action]
        msg_exit("Unsupported action #{action} specified") if method.nil?

        if METHODS_NEEDING_DATA.include?(method)
          data = opts[:inputfile].empty? ? prompt_get_data : File.read(opts[:inputfile])

          msg_exit("Action #{action} requires data to be specified") if data.empty?
        end
      end

      conn = Faraday.new(:url => opts[:url], :ssl => {:verify => false}) do |faraday|
        faraday.request(:url_encoded)               # form-encode POST params
        faraday.response(:logger) if opts[:verbose] # log requests to STDOUT
        faraday.use FaradayMiddleware::FollowRedirects, :limit => 3, :standards_compliant => true
        faraday.adapter(Faraday.default_adapter)    # make requests with Net::HTTP
        faraday.basic_auth(opts[:user], opts[:password]) if opts[:token].empty? && opts[:miqtoken].empty?
      end

      if action == "run"
        puts "Loading #{api_script}"
        require api_script
        as = ApiScript.new(CTYPE, conn)
        puts "Running #{api_script} with method #{method} ..."
        method.nil? ? as.run : as.run(method)
        exit
      end

      path = PREFIX.dup
      path << "/v#{opts[:apiversion]}" unless opts[:apiversion].empty?

      collection = ""
      item = ""
      unless resource.nil?
        path << resource
        rscan = resource.scan(%r{[^/]+})
        collection, item = rscan[0..1]
      end

      if opts[:verbose]
        puts SEP
        puts "Connection Endpoint: #{opts[:url]}"
        puts "Action:              #{action}"
        puts "HTTP Method:         #{method}"
        puts "Resource:            #{resource}"
        puts "Collection:          #{collection}"
        puts "Item:                #{item}"
        puts "Parameters:"
        params.keys.each { |k| puts "#{' ' * 21}#{k} = #{params[k]}" }
        puts "Path:                #{path}"
        puts "Data:                #{data}"
      end

      begin
        response = conn.run_request(method.to_sym, path, nil, nil) do |req|
          req.headers[:content_type]  = CTYPE
          req.headers[:accept]        = CTYPE
          req.headers['X-MIQ-Group']  = opts[:group] unless opts[:group].empty?
          req.headers['X-MIQ-Token']  = opts[:miqtoken] unless opts[:miqtoken].empty?
          req.headers['X-Auth-Token'] = opts[:token] unless opts[:token].empty?
          req.params.merge!(params)
          req.body = data if METHODS_NEEDING_DATA.include?(method)
        end
      rescue => e
        msg_exit("\nFailed to connect to #{opts[:url]} - #{e}")
      end

      if opts[:verbose]
        puts SEP
        puts "Response Headers:"
        puts response.headers

        puts SEP
        puts "Response Body:"
      end

      if response.body
        body = response.body.strip
        if opts[:format] == "pretty"
          puts json_pretty(body) unless body.empty?
        else
          puts body
        end
      end

      exit response.status >= 400 ? 1 : 0
    end
  end
end
RestApi::Cli.new.run


text:
from __future__ import absolute_import, print_function, unicode_literals

from .conf import settings
from .exceptions import UUIDNotFoundError
from .utils import get_uuid, load_tree, passthrough_loader

_STATIONS_MAP = load_tree(settings['STATIONS_MAP_DIR'],
                          data_loader=passthrough_loader)

def get_station_map(uuid=None):
    """Get the entire station map, given a UUID

    Usage::
        >>> from openkongqi.stations import get_station_map
        >>> get_uuid_map('cn-shanghai')
        ...

    :param uuid: a UUID key
    :type uuid: str
    """
    try:
        id_map = _STATIONS_MAP[uuid]
    except KeyError:
        raise UUIDNotFoundError("Unknown UUID ({})".format(uuid))
    return id_map

def get_all_uuids():
    """Return a sorted list of complete station UUIDs."""
    res = []
    for country, station_map in _STATIONS_MAP.items():
        res.extend([get_uuid(country, s) for s in station_map.values()])
    return sorted(res)


text:
[![Build Status](https://travis-ci.org/SciRuby/statsample.svg?branch=master)](https://travis-ci.org/SciRuby/statsample)
[![Code Climate](https://codeclimate.com/github/SciRuby/statsample/badges/gpa.svg)](https://codeclimate.com/github/SciRuby/statsample)
[![Gem Version](https://badge.fury.io/rb/statsample.svg)](http://badge.fury.io/rb/statsample)

Homepage :: https://github.com/sciruby/statsample

# Installation

You should have a recent version of GSL and R (with the `irr` and `Rserve` libraries) installed. In Ubuntu:

```bash
$ sudo apt-get install libgsl0-dev r-base r-base-dev
$ sudo Rscript -e "install.packages(c('Rserve', 'irr'))"
```

With these libraries in place, just install from rubygems:

```bash
$ [sudo] gem install statsample
```

On *nix, you should install statsample-optimization to retrieve gems gsl, statistics2 and a C extension to speed some methods.

```bash
$ [sudo] gem install statsample-optimization
```

If you need to work on Structural Equation Modeling, you could see +statsample-sem+. You need R with +sem+ or +OpenMx+ [http://openmx.psyc.virginia.edu/] libraries installed

```bash
$ [sudo] gem install statsample-sem
```
# Testing

See CONTRIBUTING for information on testing and contributing to statsample.

# Documentation

You can see the latest documentation in [rubydoc.info](http://www.rubydoc.info/github/sciruby/statsample/master).

# Usage

## Notebooks

You can see some iruby notebooks here:

### Statistics

* [Correlation Matrix with daru and statsample](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Statistics/Correlation%20Matrix%20with%20daru%20and%20statsample.ipynb)
* [Dominance Analysis with statsample](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Statistics/Dominance%20Analysis%20with%20statsample.ipynb)
* [Reliability ICC](http://nbviewer.ipython.org/github/v0dro/sciruby-notebooks/blob/master/Statistics/Reliability%20ICC%20with%20statsample.ipynb)
* [Levene Test](http://nbviewer.ipython.org/github/v0dro/sciruby-notebooks/blob/master/Statistics/Levene%20Test.ipynb)
* [Multiple Regression](http://nbviewer.ipython.org/github/v0dro/sciruby-notebooks/blob/master/Statistics/Multiple%20Regression.ipynb)
* [Parallel Analysis on PCA](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Statistics/Parallel%20Analysis%20on%20PCA.ipynb)
* [Polychoric Analysis](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Statistics/Polychoric%20Correlation.ipynb)
* [Reliability Scale and Multiscale Analysis](https://github.com/SciRuby/sciruby-notebooks/blob/master/Statistics/Reliability%20Scale%20Analysis.ipynb)
* [Velicer MAP Test](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Statistics/Velicer%20MAP%20test.ipynb)

### Visualizations

* [Creating Boxplots with daru and statsample](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Visualization/Boxplot%20with%20daru%20and%20statsample.ipynb)
* [Creating A Histogram](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Visualization/Creating%20a%20Histogram.ipynb)
* [Creating a Scatterplot](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Visualization/Scatterplot%20with%20statsample.ipynb)

### Working with DataFrame and Vector

* [Creating Vectors and DataFrames with daru](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Creation%20of%20Vector%20and%20DataFrame.ipynb)
* [Detailed Usage of Daru::Vector](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Usage%20of%20Vector.ipynb)
* [Detailed Usage of Daru::DataFrame](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Data%20Analysis/Usage%20of%20DataFrame.ipynb)
* [Visualizing Data with Daru::DataFrame](http://nbviewer.ipython.org/github/SciRuby/sciruby-notebooks/blob/master/Visualization/Visualizing%20data%20with%20daru%20DataFrame.ipynb)

## Examples

See the /examples directory for some use cases. The notebooks listed above have mostly
the same examples, and they look better so you might want to see that first.

# Description

A suite for basic and advanced statistics on Ruby. Tested on CRuby 2.0.0, 2.1.1, 2.2 and 2.3.0 See `.travis.yml` for more information.

Include:
- Descriptive statistics: frequencies, median, mean, standard error, skew, kurtosis (and many others).
- Correlations: Pearson's r, Spearman's rank correlation (rho), point biserial, tau a, tau b and  gamma.  Tetrachoric and Polychoric correlation provides by +statsample-bivariate-extension+ gem.
- Intra-class correlation
- Anova: generic and vector-based One-way ANOVA and Two-way ANOVA, with contrasts for One-way ANOVA.
- Tests: F, T, Levene, U-Mannwhitney.
- Regression: Simple, Multiple (OLS)
- Factorial Analysis: Extraction (PCA and Principal Axis), Rotation (Varimax, Equimax, Quartimax) and Parallel Analysis and Velicer's MAP test, for estimation of number of factors.
- Reliability analysis for simple scale and a DSL to easily analyze multiple scales using factor analysis and correlations, if you want it.
- Basic time series support
- Dominance Analysis, with multivariate dependent and bootstrap (Azen & Budescu)
- Sample calculation related formulas
- Structural Equation Modeling (SEM), using R libraries +sem+ and +OpenMx+
- Creates reports on text, html and rtf, using ReportBuilder gem
- Graphics: Histogram, Boxplot and Scatterplot

## Principles

- Software Design:
  - One module/class for each type of analysis
  - Options can be set as hash on initialize() or as setters methods
  - Clean API for interactive sessions
  - summary() returns all necessary informacion for interactive sessions
  - All statistical data available though methods on objects
  - All (important) methods should be tested. Better with random data.
- Statistical Design
  - Results are tested against text results, SPSS and R outputs.
  - Go beyond Null Hiphotesis Testing, using confidence intervals and effect sizes when possible
  - (When possible) All references for methods are documented, providing sensible information on documentation

# Features

- Classes for manipulation and storage of data:
  - Uses [daru](https://github.com/v0dro/daru) for storing data and basic statistics.
  - Statsample::Multiset: multiple datasets with same fields and type of vectors
- Anova module provides generic Statsample::Anova::OneWay and vector based Statsample::Anova::OneWayWithVectors. Also you can create contrast using Statsample::Anova::Contrast
- Module Statsample::Bivariate provides covariance and pearson, spearman, point biserial, tau a, tau b, gamma, tetrachoric (see Bivariate::Tetrachoric) and polychoric (see Bivariate::Polychoric) correlations. Include methods to create correlation and covariance matrices
- Multiple types of regression.
  - Simple Regression :  Statsample::Regression::Simple
  - Multiple Regression: Statsample::Regression::Multiple
- Factorial Analysis algorithms on Statsample::Factor module.
  - Classes for Extraction of factors:
    - Statsample::Factor::PCA
    - Statsample::Factor::PrincipalAxis
  - Classes for Rotation of factors:
    - Statsample::Factor::Varimax
    - Statsample::Factor::Equimax
    - Statsample::Factor::Quartimax
  - Classes for calculation of factors to retain
    - Statsample::Factor::ParallelAnalysis performs Horn's 'parallel analysis' to a principal components analysis to adjust for sample bias in the retention of components.
    - Statsample::Factor::MAP performs Velicer's Minimum Average Partial (MAP) test, which retain components as long as the variance in the correlation matrix represents systematic variance.
- Dominance Analysis. Based on Budescu and Azen papers, dominance analysis is a method to analyze the relative importance of one predictor relative to another on multiple regression
  - Statsample::DominanceAnalysis class can report dominance analysis for a sample, using uni or multivariate dependent variables
  - Statsample::DominanceAnalysis::Bootstrap can execute bootstrap analysis to determine dominance stability, as recomended by  Azen & Budescu (2003) link[http://psycnet.apa.org/journals/met/8/2/129/].
- Module Statsample::Codification, to help to codify open questions
- Converters to export data:
  - Statsample::Mx    : Write Mx Files
  - Statsample::GGobi : Write Ggobi files
- Module Statsample::Crosstab provides function to create crosstab for categorical data
- Module Statsample::Reliability provides functions to analyze scales with psychometric methods.
  - Class Statsample::Reliability::ScaleAnalysis provides statistics like mean, standard deviation for a scale, Cronbach's alpha and standarized Cronbach's alpha, and for each item: mean, correlation with total scale, mean if deleted, Cronbach's alpha is deleted.
  - Class Statsample::Reliability::MultiScaleAnalysis provides a DSL to easily analyze reliability of multiple scales and retrieve correlation matrix and factor analysis of them.
  - Class Statsample::Reliability::ICC provides intra-class correlation, using Shrout & Fleiss(1979) and McGraw & Wong (1996) formulations.
- Module Statsample::SRS (Simple Random Sampling) provides a lot of functions to estimate standard error for several type of samples
- Module Statsample::Test provides several methods and classes to perform inferencial statistics
  - Statsample::Test::BartlettSphericity
  - Statsample::Test::ChiSquare
  - Statsample::Test::F
  - Statsample::Test::KolmogorovSmirnov (only D value)
  - Statsample::Test::Levene
  - Statsample::Test::UMannWhitney
  - Statsample::Test::T
  - Statsample::Test::WilcoxonSignedRank
- Module Graph provides several classes to create beautiful graphs using rubyvis
  - Statsample::Graph::Boxplot
  - Statsample::Graph::Histogram
  - Statsample::Graph::Scatterplot
- Gem <tt>bio-statsample-timeseries</tt> provides module Statsample::TimeSeries with support for time series, including ARIMA estimation using Kalman-Filter.
- Gem <tt>statsample-sem</tt> provides a DSL to R libraries +sem+ and +OpenMx+
- Gem <tt>statsample-glm</tt> provides you with GML method, to work with Logistic, Poisson and Gaussian regression ,using ML or IRWLS.
- Close integration with gem <tt>reportbuilder</tt>, to easily create reports on text, html and rtf formats.

# Resources

- Source code on github :: http://github.com/sciruby/statsample
- Bug report and feature request :: http://github.com/sciruby/statsample/issues
- E-mailing list :: https://groups.google.com/forum/#!forum/sciruby-dev

# License

BSD-3 (See LICENSE.txt)

Could change between version, without previous warning. If you want a specific license, just choose the version that you need.


text:
BASELINE_VERSION=0.21.0
NEW_VERSION=0.22.0-SNAPSHOT

./apicheck.sh \
  --project=hadoop \
  --package=org.apache.hadoop \
  --baseline-version=$BASELINE_VERSION \
  --baseline-tar=http://archive.apache.org/dist/hadoop/core/hadoop-$BASELINE_VERSION/hadoop-$BASELINE_VERSION.tar.gz \
  --baseline-classpath-includes='{hadoop-{common,hdfs,mapred}-$BASELINE_VERSION.jar,lib/*.jar}' \
  --baseline-classpath-excludes='lib/hadoop-fairscheduler-$BASELINE_VERSION.jar' \
  --new-version=$NEW_VERSION \
  --new-tar=https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-22-Build/lastSuccessfulBuild/artifact/hadoop-$NEW_VERSION.tar.gz \
  --new-classpath-includes='{hadoop-{common,hdfs,mapred}-$NEW_VERSION.jar,lib/*.jar}' \
  --new-classpath-excludes='lib/hadoop-fairscheduler-$NEW_VERSION.jar' \
  --generate-excludes \
  --excluded-annotations='org.apache.hadoop.classification.InterfaceAudience$Private,org.apache.hadoop.classification.InterfaceAudience$LimitedPrivate,org.apache.hadoop.classification.InterfaceStability$Unstable,org.apache.hadoop.classification.InterfaceStability$Evolving' \
  --excludes-file=hadoop_hidden_classes \
  --hadoop-annotation-tools-jar=lib/* \
  --java-tools-classpath=$JAVA_HOME/../Classes/classes.jar:$JAVA_HOME/../Classes/jsse.jar:$JAVA_HOME/../Classes/jce.jar:$ANT_HOME/lib/ant.jar