text:
\chapter{Image I/O API Helper Classes}
\label{chap:imageioapi}
\index{Image I/O API Helper Classes|(}

\section{Data Type Descriptions: {\cf TypeDesc}}
\label{sec:dataformats}
\label{sec:TypeDesc}
\index{data formats}

There are two kinds of data that are important to \product:

\begin{itemize}
\item \emph{Internal data} is in the memory of the computer, used by an
  application program.
\item \emph{Native file data} is what is stored in an image file itself
  (i.e., on the ``other side'' of the abstraction layer that \product
  provides).
\end{itemize}

Both internal and file data is stored in a particular \emph{data format}
that describes the numerical encoding of the values.  \product
understands several types of data encodings, and there is
a special class, \TypeDesc, that allows their enumeration and
is described in the header file {\cf OpenImageIO/typedesc.h}.
A \TypeDesc describes a base data format type, aggregation into simple
vector and matrix types, and an array length (if
it's an array).

The remainder of this section describes the C++ API for \TypeDesc.
See Section~\ref{sec:pythontypedesc} for the corresponding Python
bindings.

\TypeDesc supports the following base data format types, given by the
enumerated type {\cf BASETYPE}:

\medskip

\begin{tabular}{l p{4.75in}}
{\cf UINT8} &  8-bit integer values ranging from
  0..255, corresponding to the C/C++ {\cf unsigned char}. \\
{\cf INT8} &  8-bit integer values ranging from
  -128..127, corresponding to the C/C++ {\cf char}. \\
{\cf UINT16} &  16-bit integer values ranging
  from 0..65535, corresponding to the C/C++ {\cf unsigned short}. \\
{\cf INT16} &  16-bit integer values ranging
  from -32768..32767, corresponding to the C/C++ {\cf short}. \\
{\cf UINT} &  32-bit integer values,
  corresponding to the C/C++ {\cf unsigned int}. \\
{\cf INT} &  signed 32-bit integer values, corresponding
  to the C/C++ {\cf int}. \\
{\cf UINT64} &  64-bit integer values,
  corresponding to the C/C++ {\cf unsigned long long} (on most architectures). \\
{\cf INT64} &  signed 64-bit integer values, corresponding
  to the C/C++ {\cf long long} (on most architectures). \\
{\cf FLOAT} &  32-bit IEEE floating point values,
  corresponding to the C/C++ {\cf float}. \\
{\cf DOUBLE} &  64-bit IEEE floating point values,
  corresponding to the C/C++ {\cf double}. \\
{\cf HALF} &  16-bit floating point values in the format
  supported by OpenEXR and OpenGL.
\end{tabular}
\medskip

\noindent A \TypeDesc can be constructed using just this information, either as
a single scalar value, or an array of scalar values:

\apiitem{{\ce TypeDesc} (BASETYPE btype) \\
{\ce TypeDesc} (BASETYPE btype, int arraylength)}
Construct a type description of a single scalar value of the given base
type, or an array of such scalars if an array length is supplied.  For
example, {\cf TypeDesc(UINT8)} describes an unsigned 8-bit integer,
and {\cf TypeDesc(FLOAT,7)} describes an array of 7 32-bit float values.
Note also that a non-array \TypeDesc may be implicitly constructed from
just the {\cf BASETYPE}, so it's okay to pass a {\cf BASETYPE}
to any function parameter that takes a full \TypeDesc.
\apiend

\medskip
\noindent In addition, \TypeDesc supports certain aggregate types, described
by the enumerated type {\cf AGGREGATE}:

\medskip
\begin{tabular}{l p{4.75in}}
{\cf SCALAR} & a single scalar value (such as a raw {\cf int}
  or {\cf float} in C).  This is the default. \\
{\cf VEC2} & two values representing a 2D vector. \\
{\cf VEC3} & three values representing a 3D vector. \\
{\cf VEC4} & four values representing a 4D vector. \\
{\cf MATRIX33} & nine values representing a $3 \times 3$ matrix. \\
{\cf MATRIX44} & sixteen values representing a $4 \times 4$ matrix.
\end{tabular}
\medskip

\noindent And optionally, a hint about the semantics of the data,
described by the enumerated type {\cf VECSEMANTICS}:\footnote{It's
unfortunately called {\cf VECSEMANTICS} because it used to be used
strictly for 3-vectors. If we were to do it over again, it would just
be {\cf SEMANTICS}.}

\medskip
\begin{tabular}{p{1in} p{4.25in}}
{\cf NOSEMANTICS} & nothing special known. \\
{\cf COLOR} & indicates a vector that is intended to represent
  a ``color,'' not a spatial
  quantity (and of course therefore does not undergo a transformation). \\
{\cf POINT} &  indicates a vector that represents a
  spatial position and should be transformed by a $4 \times 4$ matrix
  as if it had a 4th component of 1. \\
{\cf VECTOR} &  indicates a vector that represents a
  spatial direction and should be transformed by a $4 \times 4$ matrix
  as if it had a 4th component of 0. \\
{\cf NORMAL} &  indicates a vector that represents a
  surface normal and should be transformed like a vector, but using the
  inverse-transpose of a $4 \times 4$ matrix. \\
{\cf TIMECODE} & indicates an {\cf int[2]} representing the standard
  4-byte encoding of an SMPTE timecode. \\
{\cf KEYCODE} & indicates an {\cf int[7]} representing the standard
  28-byte encoding of an SMPTE keycode. \\
\end{tabular}
\medskip

\noindent These can be combined to fully describe a complex type:

\apiitem{{\ce TypeDesc} (BASETYPE btype, AGGREGATE agg, VECSEMANTICS
xform=NOSEMANTICS)  \\
{\ce TypeDesc} (BASETYPE btype, AGGREGATE agg, int arraylen) \\
{\ce TypeDesc} (BASETYPE btype, AGGREGATE agg, VECSEMANTICS xform, int arraylen)
}
Construct a type description of an aggregate (or array of aggregates),
with optional vector transformation semantics.  For example,
{\cf TypeDesc(HALF,COLOR)} describes an aggregate of 3 16-bit floats
comprising a color, and {\cf TypeDesc(FLOAT,VEC3,POINT)} describes
an aggregate of 3 32-bit floats comprising a 3D position.

Note that aggregates and arrays are different.  A {\cf
  TypeDesc(FLOAT,3)} is an array of three floats, a {\cf
  TypeDesc(FLOAT,COLOR)} is a single 3-channel color comprised of
floats, and {\cf TypeDesc(FLOAT,3,COLOR)} is an array of 3 color values,
each of which is comprised of 3 floats.
\apiend

\bigskip

Of these, the only ones commonly used to store pixel values in image files
are scalars of {\cf UINT8}, {\cf UINT16}, {\cf FLOAT}, and {\cf HALF}
(the last only used by OpenEXR, to the best of our knowledge).

Note that the \TypeDesc (which is also used for applications other
than images) can describe many types not used by
\product.  Please ignore this extra complexity; only the above simple types are understood by
\product as pixel storage data types, though a few others, including
{\cf STRING} and {\cf MATRIX44} aggregates, are occasionally used for
\emph{metadata} for certain image file formats (see
Sections~\ref{sec:imageoutput:metadata}, \ref{sec:imageinput:metadata},
and the documentation of individual ImageIO plugins for details).

\section{Efficient unique strings: {\cf ustring}}
\label{sec:ustring}
\indexapi{ustring}

A \ustring is an alternative to {\cf char*} or {\cf std::string} for storing
strings, in which the character sequence is stored uniquely.  If there are
many copies of the same \ustring, they all point to the same canonical
characters, which themselves are stored only once. This allows \ustring's to
be assigned to one another with only the copy of a pointer assignment (not
allocation or copying of characters), and for {\cf ==} and {\cf !=}
comparisons of two \ustring values for only the cost of comparing the
pointers (not examining the characters).

\OpenImageIO uses \ustring in a few select places, where string
assignment/copy and equality/inequality are the dominant string operation
and we want them to be the same low cost as a simple pointer assignment or
comparison.  (This primarily comes into play for the \ImageCache and
\TextureSystem, where we refer to images by their filenames and want
very fast lookups.)

Please consult the public header {\cf ustring.h} for details, especially if
you want to use \ustring extensively in your own code. But here are the most
important details to know if you are calling the \OpenImageIO functions that
take \ustring parameters:

\apiitem{{\ce ustring} (const char *chars) \\
{\ce ustring} (const char *chars, size_t length) \\
{\ce ustring} (const std::string \&str) \\
{\ce ustring} (string\_view sref)}
Constructs a \ustring from a C string ({\cf char*}), C++ {\cf std::string},
or a \stringview.
\apiend

\apiitem{const char* ustring::{\ce c_str} () \\
const std::string\& ustring::{\ce string()} \\
string\_view ustring::{\ce operator string\_view}()}
Convert a \ustring to a 0-terminated C string, a C++ {\cf std::string}, or
a {\cf string_view}.  All of these are extremely inexpensive.
\apiend

\section{Non-owning string views: {\cf string_view}}
\label{sec:stringview}
\indexapi{string_view}

A \stringview a non-owning, non-copying, non-allocating reference to a
sequence of characters.  It encapsulates both a character pointer and a
length.

A function that takes a string input (but does not need to alter the string
in place) may use a \stringview parameter and accept input that is any of
{\cf char*} (C string), string literal (constant char array), a {\cf
std::string} (C++ string), or OIIO \ustring.  For all of these cases, no
extra allocations are performed, and no extra copies of the string  contents
are performed (as they would be, for example, if the function took a const
{\cf std::string\&} argument but was passed a {\cf char*} or string
literal).

Furthermore, a function that returns a copy or a substring of one of its
inputs (for example, a {\cf substr()}-like function) may return a \stringview
rather than a {\cf std::string}, and thus generate its return value without
any allocation or copying. Upon assignment to a {\cf std::string} or \ustring,
it will properly auto-convert.

There are two important caveats to using this class:
\begin{enumerate}
\item The \stringview merely refers to characters owned by another string,
   so the \stringview may not be used outside the lifetime of the string
   it refers to. Thus, \stringview is great for parameter passing, but
   it's not a good idea to use a \stringview to store strings in a data
   structure (unless you are really sure you know what you're doing).
\item Because the run of characters that the \stringview refers to may not
   be 0-terminated, it is important to distinguish between the {\cf data()}
   method, which returns the pointer to the characters, and the {\cf c_str()}
   method, which is guaranteed to return a valid C string that is
   0-terminated. Thus, if you want to pass the contents of a \stringview
   to a function that expects a 0-terminated string (say, fopen), you
   must call {\cf fopen(my_string_view.c_str()}).  Note that the usual case
   is that the \stringview does refer to a 0-terminated string, and in
   that case c_str() returns the same thing as data() without any extra
   expense; but in the rare case that it is not 0-terminated, c_str()
   will incur extra expense to internally allocate a valid C string.
\end{enumerate}

\apiitem{{\ce string_view} (const char *chars) \\
{\ce string_view} (const char *chars, size_t length) \\
{\ce string_view} (const std::string \&str) \\
{\ce string_view} (ustring ustr)}
Constructs a \stringview.  The \stringview doesn't have its own copy of the
characters, so don't use the \stringview after the original string has been
destroyed or altered.

Note that the version that takes a {\cf const char*} but not a length will
automatically take the {\cf strlen(chars)} to determine the length.  (All
the other constructors can deduce the length without walking through all
of the characters.)
\apiend

\apiitem{const char* string_view::{\ce data} () \\
size_t string_view::{\ce size} ()}
The raw pointer to the characters (not necessarily 0-terminated!)
and the length of the \stringview.
\apiend

\apiitem{const char* string_view::{\ce c_str} ()}
Return a 0-terminated {\cf char*} version of the \stringview (a proper C
string).
\apiend

\apiitem{std::string (string_view sr) \\
ustring (string_view sr)}
Automatic constructions of C++ {\cf std::string} or OIIO \ustring from
a \stringview.
\apiend

\smallskip
\noindent Additionally, a large portion of the usual API for {\cf std::string}
is mimicked by \stringview.  Please consult the public {\cf string_view.h}
header file for full details, if you wish to use \stringview in your own
code.

\section{Non-owning array views: {\cf array_view}}
\label{sec:arrayview}
\indexapi{array_view}

A {\cf template array\_view<typename T>} is a non-owning, non-copying, non-
allocating reference to an array.  It encapsulates both a pointer and a
length, and thus is a safer way of passing pointers around (because the
function called knows how long the array is). A function that might
ordinarily take a {\cf T*} and a length could instead just take an {\cf
array\_view<T>}.

An \arrayview may be initialized explicitly from a pointer and length, by
initializing with a {\cf std::vector<T>}, or by initalizing with a constant
(treated as an array of length 1). For all of these cases, no extra
allocations are performed, and no extra copies of the array contents are
made.

Important caveat: The \arrayview merely refers to items owned by another
array, so the \arrayview should not be used outside the lifetime of the
array it refers to. Thus, \arrayview is great for parameter passing, but
it's not a good idea to use a \arrayview to store strings in a data
structure (unless you are really sure you know what you're doing).

\noindent Commonly used \arrayview methods include:

\apiitem{{\ce array_view<T>} (T *data, size_t len) \\
{\ce array_view<T>} (T \&data) \\
{\ce array_view<T>} (T *begin, T *end) \\
{\ce array_view<T>} (std::vector<T> \&vec)}
Constructs a \arrayview.  The \arrayview doesn't have its own copy of the
array elements, so don't use the \arrayview after the original array has been
destroyed or altered.
\apiend

\apiitem{T* array_view<T>::{\ce data} () \\
size_t array_view<T>::{\ce size} ()}
The raw pointer to the array, and its length.
\apiend

\apiitem{T\& array_view<T>::operator{\ce []} (size_t pos)}
References a single element of the \arrayview.
\apiend

\smallskip
\noindent Please consult the public {\cf array_view.h}
header file for full details, if you wish to use \arrayview in your own
code.

\section{Image Specification: {\cf ImageSpec}}
\label{sec:ImageSpec}
\indexapi{ImageSpec}

An \ImageSpec is a structure that describes the complete
format specification of a single image.  It contains:

\begin{itemize}
\item The image resolution (number of pixels) and origin. This specifies
  what is often called the ``pixel data window.''
\item The full size and offset of an abstract ``full'' or ``display''
  window. Differing full and data windows can indicate that the pixels
  are a crop region or a larger image, or contain overscan pixels.
\item Whether the image is organized into \emph{tiles}, and if so, the
  tile size.
\item The \emph{native data format} of the pixel values (e.g., float, 8-bit
  integer, etc.).
\item The number of color channels in the image (e.g., 3 for RGB
  images), names of the channels, and whether any particular channels
  represent \emph{alpha} and \emph{depth}.
\item A user-extensible (and format-extensible) list of any other
  arbitrarily-named and -typed data that may help describe the image or
  its disk representation.
\end{itemize}

The remainder of this section describes the C++ API for \ImageSpec.
See Section~\ref{sec:pythonimagespec} for the corresponding Python
bindings.

\subsection{\ImageSpec Data Members}

The \ImageSpec contains data fields for the values that are
required to describe nearly any image, and an extensible list of
arbitrary attributes that can hold metadata that may be user-defined or
specific to individual file formats.  Here are the hard-coded data
fields:

\apiitem{int width, height, depth \\
int x, y, z}

{\cf width, height, depth} are the size of the data of this image, i.e.,
the number of pixels in each dimension.  A {\cf depth} greater than 1
indicates a 3D ``volumetric'' image.

{\cf x, y, z} indicate the \emph{origin} of the pixel data of the image.
These default to (0,0,0), but setting them differently may indicate that
this image is offset from the usual origin.

Therefore the pixel data are defined over pixel coordinates
[{\cf x} ... {\cf x+width-1}] horizontally,
[{\cf y} ... {\cf y+height-1}] vertically,
and [{\cf z} ... {\cf z+depth-1}] in depth.
\apiend

\apiitem{int full_width, full_height, full_depth \\
int full_x, full_y, full_z}

These fields define a ``full'' or ``display'' image window over the
region [{\cf full_x} ... {\cf full_x+full_width-1}] horizontally,
[{\cf full_y} ... {\cf full_y+full_height-1}] vertically,
and [{\cf full_z} ... {\cf full_z+full_depth-1}] in depth.

Having the full display window different from the pixel data window can
be helpful in cases where you want to indicate that your image is a
\emph{crop window} of a larger image (if the pixel data window is a
subset of the full display window), or that the pixels include
\emph{overscan} (if the pixel data is a superset of the full display
window), or may simply indicate how different non-overlapping images
piece together.
\apiend

\apiitem{int tile_width, tile_height, tile_depth}
If nonzero, indicates that the image is stored on disk organized into
rectangular \emph{tiles} of the given dimension.  The default of
(0,0,0) indicates that the image is stored in scanline order, rather
than as tiles.
\apiend

\apiitem{int nchannels}
The number of \emph{channels} (color values) present in each pixel of
the image.  For example, an RGB image has 3 channels.
\apiend

%\newpage
%\vspace{24pt}  % make some space

\apiitem{TypeDesc format \\[0.5ex]
std::vector<TypeDesc> channelformats}
Describes the native format of the pixel data values themselves, as a
\TypeDesc (see \ref{sec:TypeDesc}).  Typical values would be
{\cf TypeDesc::UINT8} for 8-bit unsigned values, {\cf TypeDesc::FLOAT}
for 32-bit floating-point values, etc.

If all channels of the image have the same data format, that will
be described by {\cf format} and {\cf channelformats} will be empty
(zero length).

If there are different data formats for each channel, they will be
described in the {\cf channelformats} vector, and the {\cf format} field
will indicate a single default data format for applications that don't
wish to support per-channel formats (usually this will be the format
of the channel that has the most precision).
\apiend

\apiitem{std::vector<std::string> channelnames}
The names of each channel, in order.  Typically this will be \qkw{R},
\qkw{G},\qkw{B}, \qkw{A} (alpha), \qkw{Z} (depth), or other arbitrary
names.
\apiend

\apiitem{int alpha_channel}
The index of the channel that represents \emph{alpha} (pixel coverage
and/or transparency).  It defaults to -1 if no alpha channel is present,
or if it is not known which channel represents alpha.
\apiend

\apiitem{int z_channel}
The index of the channel that respresents \emph{z} or \emph{depth} (from
the camera).  It defaults to -1 if no depth channel is present, or if it
is not know which channel represents depth.
\apiend

\apiitem{bool deep}
If {\cf true}, this indicates that the image describes contains ``deep''
data consisting of multiple samples per pixel.  If {\cf false}, it's an
ordinary image with one data value (per channel) per pixel.
\apiend

\apiitem{ParamValueList extra_attribs}
A list of arbitrarily-named and arbitrarily-typed additional attributes
of the image, for any metadata not described by the hard-coded fields
described above.  This list may be manipulated with the {\cf
attribute()} and {\cf find_attribute()} methods.
\apiend

\subsection{\ImageSpec member functions}
\label{sec:ImageSpecMemberFuncs}

\noindent \ImageSpec contains the following methods that
manipulate format specs or compute useful information about images given
their format spec:

\apiitem{{\ce ImageSpec} (int xres, int yres, int nchans, TypeDesc fmt = UINT8)}
Constructs an \ImageSpec with the given $x$ and $y$ resolution, number
of channels, and pixel data format.

All other fields are set to the obvious defaults -- the image is an
ordinary 2D image (not a volume), the image is not offset or a crop of a
bigger image, the image is scanline-oriented (not tiled), channel names
are ``R'', ``G'', ``B,'' and ``A'' (up to and including 4 channels,
beyond that they are named ``channel \emph{n}''), the fourth channel (if
it exists) is assumed to be alpha, and values are assumed to be linear.
\apiend

\apiitem{void {\ce set_format} (TypeDesc fmt)}
Sets the format as described, and clears any per-channel format information
in {\cf channelformats}.
\apiend

\apiitem{void {\ce default_channel_names} ()}
Sets the {\cf channelnames} to reasonable defaults for the number of
channels.  Specifically, channel names are set to ``R'', ``G'', ``B,''
and ``A'' (up to and including 4 channels, beyond that they are named
``channel\emph{n}''.
\apiend

\apiitem{size_t {\ce channel_bytes} () const}
Returns the number of bytes comprising each channel of each pixel (i.e.,
the size of a single value of the type described by the {\cf format} field).
\apiend

\apiitem{size_t {\ce channel_bytes} (int chan, bool native=false) const}
Returns the number of bytes needed for the single specified
channel.  If native is {\cf false} (default), compute the size of one
channel of {\cf this->format}, but if native is {\cf true}, compute the size
of the channel in terms of the ``native'' data format of that
channel as stored in the file.
\apiend

\apiitem{size_t {\ce pixel_bytes} (bool native=false) const}
Returns the number of bytes comprising each pixel (i.e. the number of
channels multiplied by the channel size).

If {\cf native} is true, this will be the sum of all the per-channel
formats in {\cf channelformats}.  If {\cf native} is false (the
default), or if all channels use the same format, this will simply be
the number of channels multiplied by the width (in bytes) of the {\cf format}.
\apiend

\apiitem{size_t {\ce pixel_bytes} (int chbegin, int chend, bool native=false) const}
Returns the number of bytes comprising the range of channels
  $[${\cf chbegin}, {\cf chend}$)$ for each pixel.

If {\cf native} is true, this will be the sum of the per-channel
formats in {\cf channelformats} (for the given range of channels).
If {\cf native} is false (the
default), or if all channels use the same format, this will simply be
the number of channels multiplied by the width (in bytes) of the {\cf format}.
\apiend

\apiitem{imagesize_t {\ce scanline_bytes} (bool native=false) const}
Returns the number of bytes comprising each scanline,
i.e., \\ {\cf pixel_bytes(native) * width} \\
This will return {\cf std::numeric_limits<imagesize_t>::max()} in the event
of an overflow where it's not representable in an {\cf imagesize_t}.
\apiend

\apiitem{imagesize_t {\ce tile_pixels} () const}
Returns the number of tiles comprising an image tile (if it's a tiled image).
This will return {\cf std::numeric_limits<imagesize_t>::max()} in the event
of an overflow where it's not representable in an {\cf imagesize_t}.
\apiend

\apiitem{imagesize_t {\ce tile_bytes} (bool native=false) const}
Returns the number of bytes comprising an image tile (if it's a tiled
image), i.e., \\ {\cf pixel_bytes(native) * tile_width * tile_height * tile_depth } \\
This will return {\cf std::numeric_limits<imagesize_t>::max()} in the event
of an overflow where it's not representable in an {\cf imagesize_t}.
\apiend

\apiitem{imagesize_t {\ce image_pixels} () const}
Returns the number of pixels comprising an entire image image of these dimensions.
This will return {\cf std::numeric_limits<imagesize_t>::max()} in the event
of an overflow where it's not representable in an {\cf imagesize_t}.
\apiend

\apiitem{imagesize_t {\ce image_bytes} (bool native=false) const}
Returns the number of bytes comprising an entire image of these
dimensions, i.e., \\
{\cf pixel_bytes(native) * width * height * depth } \\
This will return {\cf std::numeric_limits<imagesize_t>::max()} in the event
of an overflow where it's not representable in an {\cf imagesize_t}.
\apiend

\apiitem{bool {\ce size_t_safe} () const}
Return {\cf true} if an image described by this spec can the sizes
(in pixels or bytes) of its scanlines, tiles, and the entire image can
be represented by a {\cf size_t} on that platform.  If this returns
{\cf false}, the client application should be very careful allocating
storage!
\apiend

% FIXME - document auto_stride() ?

\apiitem{void {\ce attribute} (string_view name, TypeDesc type, \\
\bigspc const void *value)}
Add a metadata attribute to {\cf extra_attribs}, with the given name and
data type.  The {\cf value} pointer specifies
the address of the data to be copied.
\apiend

\apiitem{void {\ce attribute} (string_view name, unsigned int value)\\
    void {\ce attribute} (string_view name, int value)\\
    void {\ce attribute} (string_view name, float value)\\
    void {\ce attribute} (string_view name, string_view value)}
Shortcuts for passing attributes comprised of a single integer,
floating-point value, or string.
\apiend

\apiitem{void {\ce erase_attribute} (string_view name,\\
\bigspc\bigspc                   TypeDesc searchtype=UNKNOWN,\\
\bigspc\bigspc                   bool casesensitive=false)}

Searches {\cf extra_attribs} for an attribute matching {\cf name},
and if it exists, remove it entirely from {\cf extra_attribs}.
If {\cf searchtype} is anything other than {\cf TypeDesc::UNKNOWN},
matches will be restricted only to attributes with the given type.
The name comparison will be exact if {\cf casesensitive} is true, otherwise
in a case-insensitive manner if {\cf caseinsensitive} is false.
\apiend

\apiitem{ImageIOParameter * {\ce find_attribute} (string_view name,\\
\bigspc\bigspc\bigspc                        TypeDesc searchtype=UNKNOWN,\\
\bigspc\bigspc\bigspc                        bool casesensitive=false)\\
const ImageIOParameter * {\ce find_attribute} (string_view name,\\
\bigspc\bigspc\bigspc\spc                     TypeDesc searchtype=UNKNOWN,\\
\bigspc\bigspc\bigspc\spc                     bool casesensitive=false) const}

Searches {\cf extra_attribs} for an attribute matching {\cf name},
returning a pointer to the attribute record, or NULL if there was no
match.  If {\cf searchtype} is anything other than {\cf TypeDesc::UNKNOWN},
matches will be restricted only to attributes with the given type.
The name comparison will be exact if {\cf casesensitive} is true, otherwise
in a case-insensitive manner if {\cf caseinsensitive} is false.
\apiend

\apiitem{const ImageIOParameter * {\ce find_attribute} (string_view name,\\
\bigspc\bigspc\bigspc\spc                     ImageIOParameter \&tmpparam,\\
\bigspc\bigspc\bigspc\spc                     TypeDesc searchtype=UNKNOWN,\\
\bigspc\bigspc\bigspc\spc                     bool casesensitive=false) const}
Search for the named attribute and return the pointer to its
{\cf ImageIOParameter} record, or {\cf NULL} if not found.  This variety of
{\cf find_attribute(}) can retrieve items such as \qkw{width}, which are
part of the \ImageSpec, but not in {\cf extra_attribs}. The {\cf tmpparam}
is a temporary storage area owned by the caller, which is used as temporary
buffer in cases where the information does not correspond to an actual
{\cf extra_attribs} (in this case, the return value will be {\cf \&tmpparam}).
\apiend

\apiitem{int {\ce get_int_attribute} (string_view name, int defaultval=0) const}
Gets an integer metadata attribute (silently converting to {\cf int}
even if if the data is really int8, uint8, int16, uint16, or uint32),
and simply substituting the supplied default value if no such metadata
exists.  This is a convenience function for when you know you are just
looking for a simple integer value.
\apiend

\apiitem{float {\ce get_float_attribute} (string_view name, float defaultval=0) const}
Gets a float metadata attribute (silently converting to {\cf float} even
if the data is really half or double), simply substituting the supplied
default value if no such metadata exists.  This is a convenience
function for when you know you are just looking for a simple float value.
\apiend

\apiitem{string_view {\ce get_string_attribute} (string_view name, \\
\bigspc\bigspc\bigspc  string_view defaultval = "") const}
Gets a string metadata attribute, simply substituting the supplied
default value if no such metadata exists.  This is a convenience
function for when you know you are just looking for a simple string value.
\apiend

\apiitem{static std::string {\ce metadata_val} (const ImageIOParamaeter \&p,
  bool human=true) const}
For a given parameter {\cf p}, format the value nicely as a string.
If {\cf human} is true, use
especially human-readable explanations (units, or decoding of
values) for certain known metadata.
\apiend

\apiitem{std::string {\ce to_xml} () const}
Saves the contents of the \ImageSpec as XML, returning it as a string.
\apiend

\apiitem{void {\ce from_xml} (const char *xml) const}
Populates the fields of the \ImageSpec based on the XML passed in.
\apiend

\apiitem{TypeDesc {\ce channelformat} (int chan) const}
Returns a \TypeDesc describing the format of the requested channel.
\apiend

\apiitem{void {\ce get_channelformats} (std::vector<TypeDesc> \&formats) const}
Fill in an array of channel formats describing all channels in the
image.  (Note that this differs slightly from the member data
{\cf channelformats}, which is empty if there are not separate per-channel
formats.)
\apiend

\section{``Deep'' pixel data: {\cf DeepData}}
\label{sec:deepdata}
\indexapi{DeepData}

A \DeepData holds the contents of an image of ``deep'' pixels (multiple
depth samples per pixel).

\noindent Commonly used \DeepData fields and methods include:

\apiitem{void {\ce init} (const ImageSpec \&spec)}
Initialize the \DeepData based on the \ImageSpec's total number of pixels,
number and types of channels. At this stage, all pixels are assumed to
have 0 samples, and no sample data is allocated.
\apiend

\apiitem {void {\ce init} (int npix, int nchan,\\
    \bigspc array_view<const TypeDesc> channeltypes),\\
    \bigspc array_view<const std::string> channelnames)}
Initialize the \DeepData with a number of pixels, channels, channel
types, and channel names.
\apiend

\apiitem{void {\ce clear} ()}
Reset the \DeepData to be equivalent to its empty initial state.
\apiend

\apiitem{void {\ce free} ()}
Not only {\cf clear()}, but also ensure that all allocated memory has been
truly freed.
\apiend

\apiitem{int {\ce npixels} () const}
Retrieve the total number of pixels.
\apiend

\apiitem{int {\ce nchannels} () const}
Retrieve the number of channels.
\apiend

\apiitem{string_view {\ce channelname} (int c) const}
\NEW % 1.7
Retrieve the name of channel {\cf c}.
\apiend

\apiitem{TypeDesc {\ce channeltype} (int c) const}
Retrieve the data type of channel {\cf c}.
\apiend

\apiitem{size_t {\ce channelsize} (int c) const}
Retrieve the size (in bytes) of one datum of channel {\cf c}.
\apiend

\apiitem{size_t {\ce samplesize} () const}
Retrieve the packed size (in bytes) of all channels of one sample.
\apiend

\apiitem{int {\ce samples} (int pixel) const}
Retrieve the number of samples for the given pixel index.
\apiend

\apiitem{void {\ce set_samples} (int pixel, int samps)}
Set the number of samples for the given pixel index.
\apiend

\apiitem{void {\ce insert_samples} (int pixel, int samplepos, int n=1)}
\NEW % 1.7
Insert {\cf n} samples of the specified pixel, betinning at the sample
position index. After insertion, the new samples will have uninitialized
values.
\apiend

\apiitem{void {\ce erase_samples} (int pixel, int samplepos, int n=1)}
\NEW % 1.7
Remove {\cf n} samples of the specified pixel, betinning at the sample
position index.
\apiend

\apiitem{float {\ce deep_value} (int pixel, int channel, int sample) const \\
uint32_t {\ce deep_value_uint} (int pixel, int channel, int sample) const}
Retrieve the value of the given pixel, channel, and sample indices, for
floating point or unsigned integer channel types, respectively.
\apiend

\apiitem{void {\ce set_deep_value} (int pixel, int channel, int sample, float value) \\
void {\ce set_deep_value} (int pixel, int channel, int sample, uint32_t value)}
Set the value of the given pixel, channel, and sample indices, for
floating point or unsigned integer channel types, respectively.
\apiend

\apiitem{bool {\ce copy_deep_sample} (int pixel, int sample, \\
    \bigspc const DeepData \&src, int srcpixel, int srcsample)}
\NEW % 1.7
Copy a deep sample from {\cf src} to this \DeepData. They must have the same
channel layout. Return {\cf true} if ok, {\cf false} if the operation could
not be performed.
\apiend

\apiitem{bool {\ce copy_deep_pixel} (int pixel, const DeepData \&src, int srcpixel)}
\NEW % 1.7
Copy an entire deep pixel from {\cf src} to this \DeepData, completely
replacing any pixel data for that pixel. They must have the same channel
layout. Return {\cf true} if ok, {\cf false} if the operation could not be
performed.
\apiend

\apiitem{void {\ce split} (int pixel, float depth)}
\NEW % 1.7
Split any samples of the pixel that cross {\cf depth}.
\apiend

\apiitem{void {\ce sort} (int pixel)}
\NEW % 1.7
Sort the samples of the pixel by their Z depth.
\apiend

\apiitem{void {\ce merge_overlaps} (int pixel)}
\NEW % 1.7
Merge any adjacent samples in the pixel that exactly overlap in $z$
range. This is only useful if the pixel has previously been split at
all sample starts and ends, and sorted by depth. Note that this may change
the number of samples in the pixel.
\apiend

\apiitem{bool {\ce merge_deep_pixels} (int pixel, const DeepData \&src, int srcpixel)}
\NEW % 1.7
Merge the samples of {\cf src}'s pixel into this \DeepData's pixel.
Return {\cf true} if ok, {\cf false} if the operation could not be
performed.
\apiend

\apiitem{bool {\ce occlusion_cull} (int pixel)}
\NEW % 1.7
Eliminate any samples beyond an opaque sample.
\apiend

\subsection{Miscellaneous Utilities}
\label{sec:miscapi}

These helper functions are not part of any other \OpenImageIO class,
they just exist in the {\cf OpenImageIO} namespace as general utilities.
(See Section~\ref{sec:pythonmiscapi} for the corresponding Python
bindings.)

\apiitem{int {\ce openimageio_version} ()}
\index{version}
\indexapi{openimageio_version}
Returns a numeric value for the version of \product, 10000 for each
major version, 100 for each minor version, 1 for each patch.  For
example, \product 1.2.3 would return a value of 10203.
\apiend

\apiitem{std::string {\ce geterror} ()}
\index{error checking}
\indexapi{geterror}
Returns any error string describing what went wrong if
{\cf ImageInput::create()} or \\ {\cf ImageOutput::create()} failed
(since in such cases, the \ImageInput or \ImageOutput itself does
not exist to have its own {\cf geterror()} function called).
This function returns the last error
for this particular thread; separate threads will not clobber each
other's global error messages.
\apiend

\apiitem{bool {\ce attribute} (string_view name, TypeDesc type,
  const void *val)}
\index{globalattribute}
\label{sec:globalattribute}
\indexapi{attribute}

Sets an global attribute (i.e., a property or option) of \product.
The {\cf name} designates the name of the attribute, {\cf type}
describes the type of data, and {\cf val} is a pointer to memory
containing the new value for the attribute.

If the name is known, valid attribute that matches the type specified,
the attribute will be set to the new value and {\cf attribute()} will
return {\cf true}.  If {\cf name} is not recognized, or if the types do
not match (e.g., {\cf type} is {\cf TypeDesc::TypeFloat} but the named
attribute is a string), the attribute will not be modified, and {\cf
  attribute()} will return {\cf false}.

\noindent The following are the recognized attributes:

\apiitem{int threads}
\vspace{10pt}
\index{threads} \label{sec:attribute:threads}
Some \product operations can be accelerated if allowed to spawn multiple
threads to parallelize the task.  (Examples: simultaneous format conversions
of multiple scanlines read together, or many \ImageBufAlgo operations.)
This attribute sets the default number of threads that will be spawned
for these operations (the ``fan out'').
The default is 0, which means that it should spawn as many
threads as there are hardware cores present on the system.

Situations where the main application logic is essentially single threaded
(i.e., one top-level call into OIIO at a time) should leave this at the
default value, or some reasonable number of cores, thus allowing lots of
threads to fill the cores when OIIO has big tasks to complete. But
situations where you have many threads at the application level, each of
which is expected to be making separate OIIO calls simultaneously, should
set this to 1, thus having each calling thread do its own work inside of
OIIO rather than spawning new threads with a high overall ``fan out.''
\apiend

\apiitem{int exr_threads}
\vspace{10pt}
\index{exr_threads}
Sets the internal OpenEXR thread pool size. The default is to use as many
threads as the amount of hardware concurrency detected.
Note that this is separate from the OIIO
\qkw{threads} attribute.
\apiend

\apiitem{string plugin_searchpath}
\vspace{10pt}
\index{plugin_searchpath}
A colon-separated list of directories to search for
dynamically-loaded format plugins.
\apiend

\apiitem{string format_list}
\vspace{10pt}
\index{format_list}
A comma-separated list of all the names of all supported
image format readers and writers.  (Note: can only be retrieved
by {\cf getattribute()}, cannot be set by {\cf attribute()}.)
\apiend

\apiitem{string extension_list}
\vspace{10pt}
\index{extension_list}
For each format, the format name, followed by a colon, followed by
a comma-separated list of all extensions that are presumed to be used
for that format.  Semicolons separate the lists for formats.  For
example,
\begin{code}
     "tiff:tif;jpeg:jpg,jpeg;openexr:exr"
\end{code}
(Note: can only be retrieved
by {\cf getattribute()}, cannot be set by {\cf attribute()}.)
\apiend

\apiitem{string library_list}
\vspace{10pt}
\index{library_list}
\NEW   % 1.7
For each format that uses a dependent library, the format name, followed by
a colon, followed by the name and version of the dependency. Semicolons
separate the lists for formats.  For example,
\begin{code}
     "tiff:LIBTIFF 4.0.4;gif:gif_lib 4.2.3;openexr:OpenEXR 2.2.0"
\end{code}
(Note: can only be retrieved by {\cf getattribute()}, cannot be set by
{\cf attribute()}.)
\apiend

\apiitem{int read_chunk}
\vspace{10pt}
\index{read_chunk}
When performing a {\cf read_image()}, this is the number of scanlines it
will attempt to read at a time (some formats are more efficient when reading
and decoding multiple scanlines).  The default is 256.  The special value
of 0 indicates that it should try to read the whole image if possible.
\apiend

\apiend

\apiitem{bool {\ce attribute} (string_view name, int val) \\
bool {\ce attribute} (string_view name, float val) \\
bool {\ce attribute} (string_view name, const char *val) \\
bool {\ce attribute} (string_view name, const std::string \& val)}
Specialized versions of {\cf attribute()} in which the data type is
implied by the type of the argument.
\apiend

\apiitem{bool {\ce getattribute} (string_view name, TypeDesc type,
  void *val)}
\indexapi{getattribute}

Gets the current value of a global attribute.  The {\cf name} designates
the name of the attribute, {\cf type} describes the type of data, and
{\cf val} is a pointer to memory where the user would like the value
placed.

If the attribute name is valid and matches the type specified, the
attribute value will be stored at address {\cf val} and {\cf
  attribute()} will return {\cf true}.  If {\cf name} is not recognized
as a valid attribute name, or if the types do not match (e.g., {\cf
  type} is {\cf TypeDesc::TypeFloat} but the named attribute is a
string), no data will be written to {\cf val}, and {\cf attribute()}
will return {\cf false}.

The complete list of attributes can be found above, in the description
of the {\cf attribute()} function.
\apiend

\apiitem{bool {\ce getattribute} (string_view name, int \&val) \\
bool {\ce getattribute} (string_view name, float \&val) \\
bool {\ce getattribute} (string_view name, char **val) \\
bool {\ce getattribute} (string_view name, std::string \& val)}
Specialized versions of {\cf getattribute()} in which the data type is
implied by the type of the argument.

For example, the following are equivalent to the example above for the
general (pointer) form of {\cf getattribute()}:

\begin{code}
      int threads;
      OIIO::getattribute ("threads", &threads);
      std::string path;
      OIIO::getattribute ("plugin_searchpath", &path);
\end{code}
\apiend

\apiitem{int {\ce get_int_attribute} (string_view name, int defaultvalue=0) \\
float {\ce get_float_attribute} (string_view name, float defaultvalue=0) \\
string_view {\ce get_string_attribute} (string_view name, \\
\bigspc\bigspc\bigspc  string_view defaultvalue="")}
\indexapi{get_int_attribute} \indexapi{get_float_attribute}
\indexapi{get_string_attribute}
\NEW % 1.7
Specialized versions of {\cf getattribute()} for common types, in which the
data is returned directly, and a supplied default value is returned if the
attribute was not found.

For example, the following are equivalent to the example above for the
general (pointer) form of {\cf getattribute()}:

\begin{code}
      int threads = OIIO::getattribute ("threads", 0);
      string_view path = OIIO::getattribute ("plugin_searchpath");
\end{code}
\apiend

\apiitem{void {\ce declare_imageio_format} (const std::string \&format_name,\\
\bigspc\bigspc\spc                    ImageInput::Creator input_creator, \\
\bigspc\bigspc\spc                    const char **input_extensions,\\
\bigspc\bigspc\spc                    ImageOutput::Creator output_creator, \\
\bigspc\bigspc\spc                     const char **output_extensions)}
Register the input and output `create' routines and list of file
extensions for a particular format.
\apiend

\index{Image I/O API Helper Classes|)}

\chapwidthend


text:
It lets you store all your modules into one global. It supports lazy-loading or full-loading.

* License: MIT
* Current status: alpha / unstable
* Platform: Node.js only

Full BDD spec generated by Mocha:

redpajama_set_name: RedPajamaGithub


text:
package net.sf.antcontrib.perf;

import java.io.File;
import java.io.FileWriter;
import java.io.Serializable;
import java.text.SimpleDateFormat;
import java.util.Comparator;
import java.util.Date;
import java.util.Map;
import java.util.TreeMap;
import java.util.concurrent.ConcurrentHashMap;

import org.apache.tools.ant.BuildEvent;
import org.apache.tools.ant.BuildListener;
import org.apache.tools.ant.Project;
import org.apache.tools.ant.Target;
import org.apache.tools.ant.Task;

/**
 * This BuildListener keeps track of the total time it takes for each target
 * and task to execute, then prints out the totals when the build is finished.
 * This can help pinpoint the areas where a build is taking a lot of time so
 * optimization efforts can focus where they'll do the most good. Execution
 * times are grouped by targets and tasks, and are sorted from fastest running
 * to slowest running.
 *
Output can be saved to a file by setting a property in Ant. Set
 * "performance.log" to the name of a file. This can be set either
 * on the command line with the -D option
 * (-Dperformance.log=/tmp/performance.log)
 * or in the build file itself (<property name="performance.log"
 * location="/tmp/performance.log"/>).

 *
Developed for use with Antelope, migrated to ant-contrib Oct 2003.

 *
 * @author Dale Anson
 * @version $Revision: 1.5 $
 */
public class AntPerformanceListener implements BuildListener {
    /**
     * Field targetStats.
     */
    private Map<Target, StopWatch> targetStats = new ConcurrentHashMap<Target, StopWatch>();

    /**
     * Field taskStats.
     */
    private Map<Task, StopWatch> taskStats = new ConcurrentHashMap<Task, StopWatch>();

    /**
     * Field master.
     */
    private StopWatch master = null;

    /**
     * Field swStartTime.
     */
    private long swStartTime = 0;

    /**
     * Starts a 'running total' stopwatch.
     *
     * @param be BuildEvent
     * @see org.apache.tools.ant.BuildListener#buildStarted(BuildEvent)
     */
    public void buildStarted(BuildEvent be) {
        master = new StopWatch();
        swStartTime = master.start();
    }

    /**
     * Sorts and prints the results.
     *
     * @param be BuildEvent
     * @see org.apache.tools.ant.BuildListener#buildFinished(BuildEvent)
     */
    public void buildFinished(BuildEvent be) {
        long swStopTime = master.stop();

        // sort targets, key is StopWatch, value is Target
        TreeMap<StopWatch, Target> sortedTargets = new TreeMap<StopWatch, Target>(new StopWatchComparator());
        for (Map.Entry<Target, StopWatch> entry : targetStats.entrySet()) {
            sortedTargets.put(entry.getValue(), entry.getKey());
        }

        // sort tasks, key is StopWatch, value is Task
        TreeMap<StopWatch, Task> sortedTasks = new TreeMap<StopWatch, Task>(new StopWatchComparator());
        for (Map.Entry<Task, StopWatch> entry : taskStats.entrySet()) {
            sortedTasks.put(entry.getValue(), entry.getKey());
        }

        // print the sorted results
        StringBuilder msg = new StringBuilder();
        String lSep = System.getProperty("line.separator");
        msg.append(lSep).append("Statistics:").append(lSep);
        msg.append("-------------- Target Results ---------------------").append(lSep);
        for (Map.Entry<StopWatch, Target> entry : sortedTargets.entrySet()) {
            StringBuilder sb = new StringBuilder();
            Target target = entry.getValue();
            if (target != null) {
                Project p = target.getProject();
                if (p != null && p.getName() != null) {
                    sb.append(p.getName()).append(".");
                }
                String total = format(entry.getKey().total());
                String targetName = target.getName();
                if (targetName == null || targetName.length() == 0) {
                    targetName = "<implicit>";
                }
                sb.append(targetName).append(": ").append(total);
            }
            msg.append(sb.toString()).append(lSep);
        }
        msg.append(lSep);
        msg.append("-------------- Task Results -----------------------").append(lSep);
        for (Map.Entry<StopWatch, Task> entry : sortedTasks.entrySet()) {
            Task task = entry.getValue();
            StringBuilder sb = new StringBuilder();
            Target target = task.getOwningTarget();
            if (target != null) {
                Project p = target.getProject();
                if (p != null && p.getName() != null) {
                    sb.append(p.getName()).append(".");
                }
                String targetName = target.getName();
                if (targetName == null || targetName.length() == 0) {
                    targetName = "<implicit>";
                }
                sb.append(targetName).append(".");
            }
            sb.append(task.getTaskName()).append(": ").append(format(entry.getKey().total()));
            msg.append(sb.toString()).append(lSep);
        }

        msg.append(lSep);
        msg.append("-------------- Totals -----------------------------").append(lSep);
        SimpleDateFormat format = new SimpleDateFormat("EEE, d MMM yyyy HH:mm:ss.SSS");
        msg.append("Start time: ").append(format.format(new Date(swStartTime))).append(lSep);
        msg.append("Stop time: ").append(format.format(new Date(swStopTime))).append(lSep);
        msg.append("Total time: ").append(format(master.total())).append(lSep);
        System.out.println(msg.toString());

        // write stats to file?
        Project p = be.getProject();
        File outfile = null;
        if (p != null) {
            String f = p.getProperty("performance.log");
            if (f != null) {
                outfile = new File(f);
            }
        }
        if (outfile != null) {
            try {
                FileWriter fw = new FileWriter(outfile);
                fw.write(msg.toString());
                fw.flush();
                fw.close();
                System.out.println("Wrote stats to: " + outfile.getAbsolutePath() + lSep);
            } catch (Exception e) {
                // ignored
            }
        }

        // reset the stats registers

        targetStats = new ConcurrentHashMap<Target, StopWatch>();
        taskStats = new ConcurrentHashMap<Task, StopWatch>();
    }

    /**
     * Formats the milliseconds from a StopWatch into decimal seconds.
     *
     * @param ms long
     * @return String
     */
    private String format(long ms) {
        return (ms < 1000) ? String.format("0.%03d sec", ms)
                : String.format("%d.%03d sec", ms / 1000, ms % 1000);
    }

    /**
     * Start timing the given target.
     *
     * @param be BuildEvent
     * @see org.apache.tools.ant.BuildListener#targetStarted(BuildEvent)
     */
    public void targetStarted(BuildEvent be) {
        StopWatch sw = new StopWatch();
        sw.start();
        targetStats.put(be.getTarget(), sw);
    }

    /**
     * Stop timing the given target.
     *
     * @param be BuildEvent
     * @see org.apache.tools.ant.BuildListener#targetFinished(BuildEvent)
     */
    public void targetFinished(BuildEvent be) {
        StopWatch sw = targetStats.get(be.getTarget());
        sw.stop();
    }

    /**
     * Start timing the given task.
     *
     * @param be BuildEvent
     * @see org.apache.tools.ant.BuildListener#taskStarted(BuildEvent)
     */
    public void taskStarted(BuildEvent be) {
        StopWatch sw = new StopWatch();
        sw.start();
        taskStats.put(be.getTask(), sw);
    }

    /**
     * Stop timing the given task.
     *
     * @param be BuildEvent
     * @see org.apache.tools.ant.BuildListener#taskFinished(BuildEvent)
     */
    public void taskFinished(BuildEvent be) {
        StopWatch sw = taskStats.get(be.getTask());
        if (sw != null) {
            sw.stop();
        }
    }

    /**
     * no-op.
     *
     * @param be BuildEvent
     * @see org.apache.tools.ant.BuildListener#messageLogged(BuildEvent)
     */
    public void messageLogged(BuildEvent be) {
        // does nothing
    }

    /**
     * Compares the total times for two StopWatches.
     */
    @SuppressWarnings("serial")
    public static class StopWatchComparator implements Comparator<StopWatch>, Serializable {
        /**
         * Compares the total times for two StopWatches.
         *
         * @param a StopWatch
         * @param b StopWatch
         * @return int
         */
        public int compare(StopWatch a, StopWatch b) {
            return a.total() < b.total() ? -1 : a.total() == b.total() ? 0 : 1;
        }
    }

    /**
     * A stopwatch, useful for 'quick and dirty' performance testing.
     *
     * Dale Anson
     * @version $Revision: 1.5 $
     */
    public static class StopWatch {

        /**
         * storage for start time.
         */
        private long startTime = 0;

        /**
         * storage for stop time.
         */
        private long stopTime = 0;

        /**
         * cumulative elapsed time.
         */
        private long totalTime = 0;

        /**
         * Starts the stopwatch.
         */
        public StopWatch() {
            start();
        }

        /**
         * Starts/restarts the stopwatch.
         *
         * @return the start time, the long returned System.currentTimeMillis().
         */
        public long start() {
            startTime = System.currentTimeMillis();
            return startTime;
        }

        /**
         * Stops the stopwatch.
         *
         * @return the stop time, the long returned System.currentTimeMillis().
         */
        public long stop() {
            stopTime = System.currentTimeMillis();
            totalTime += stopTime - startTime;
            startTime = 0;
            this.stopTime = 0;
            return stopTime;
        }

        /**
         * Total cumulative elapsed time.
         *
         * @return the total time
         */
        public long total() {
            return totalTime;
        }

        /**
         * Elapsed time, difference between the last start time and now.
         *
         * @return the elapsed time
         */
        public long elapsed() {
            return System.currentTimeMillis() - startTime;
        }
    }

    // quick test for the formatter

    /**
     * Method main.
     *
     * @param args String[]
     */
    public static void main(String[] args) {
        AntPerformanceListener apl = new AntPerformanceListener();

        System.out.println(apl.format(1));
        System.out.println(apl.format(10));
        System.out.println(apl.format(100));
        System.out.println(apl.format(1000));
        System.out.println(apl.format(100000));
        System.out.println(apl.format(1000000));
        System.out.println(apl.format(10000000));
        System.out.println(apl.format(100000000));
        System.out.println(apl.format(1000000000));
    }
}


text:
						RECORD TASK(S)

					 PAYROLL - START

					<form  name="form" id="form" class="form-horizontal" method="post" action="<?php echo base_url("payroll/savePayroll"); ?>" >

							Task: Time Stamp

							Time: Start

						<input type="hidden" id="hddTask" name="hddTask" value="1"/>

							Address

								<input id="viewaddress" name="viewaddress" class="form-control" type="text" disabled >
								<input id="latitud" name="latitud" type="hidden">
								<input id="longitud" name="longitud" type="hidden">
								<input id="address" name="address" type="hidden">

								<a class="btn btn-success btn-circle" href=" <?php echo base_url().'payroll/add_payroll/'; ?> ">

							Job Code/Name

								<select name="jobName" id="jobName" class="form-control" >
									<option value=''>Select...</option>
									<?php for ($i = 0; $i < count($jobs); $i++) { ?>
										<option value="<?php echo $jobs[$i]["id_job"]; ?>" ><?php echo $jobs[$i]["job_description"]; ?></option>
									<?php } ?>
								</select>

							Task/Report Description

							<textarea id="taskDescription" name="taskDescription" class="form-control" rows="3"></textarea>

								 <button type="submit" class="btn btn-primary" id='btnSubmit' name='btnSubmit'>Submit </button>

					</form>


text:
import datetime
import string
import logging
from enum import Enum
from typing import Tuple, Set, Optional, Sequence, Iterator, Iterable, TYPE_CHECKING

from colorful.fields import RGBColorField
from django.conf import settings
from django.contrib.contenttypes.fields import GenericRelation
from django.db import models
from django.db.models import Prefetch
from django.template.loader import render_to_string
from django.urls import reverse
from django.utils import timezone
from django.utils.functional import cached_property
from django.utils.html import format_html
from django.utils.translation import ugettext_lazy as _
from guardian.shortcuts import get_users_with_perms
from select2 import fields as select2_fields

from serviceform.tasks.models import Task

from .. import emails, utils
from ..utils import ColorStr

from .mixins import SubitemMixin, NameDescriptionMixin, CopyMixin
from .people import Participant, ResponsibilityPerson
from .email import EmailTemplate
from .participation import QuestionAnswer

if TYPE_CHECKING:
    from .participation import ParticipationActivity, ParticipationActivityChoice

local_tz = timezone.get_default_timezone()
logger = logging.getLogger(__name__)

class Class1:
    pass

class Class2:
    pass

def imported_symbols_anchor():
    print(RGBColorField, settings, GenericRelation, Prefetch, render_to_string, reverse, format_html,
          get_users_with_perms, select2_fields, Task, emails, CopyMixin, Participant, ResponsibilityPerson,
          EmailTemplate, QuestionAnswer, ParticipationActivity, ParticipationActivityChoice, datetime, Enum, string,
          Tuple, Set, Optional, Sequence, Iterator, Iterable, _, cached_property, models, utils, ColorStr)


text:
import unittest

from rx import Observable
from rx.testing import TestScheduler, ReactiveTest, is_prime, MockDisposable
from rx.disposables import Disposable, SerialDisposable, BooleanDisposable

on_next = ReactiveTest.on_next
on_completed = ReactiveTest.on_completed
on_error = ReactiveTest.on_error
subscribe = ReactiveTest.subscribe
subscribed = ReactiveTest.subscribed
disposed = ReactiveTest.disposed
created = ReactiveTest.created

class RxException(Exception):
    pass

# Helper function for raising exceptions within lambdas
def _raise(ex):
    raise RxException(ex)

class TestFromArray(unittest.TestCase):
    def test_subscribe_to_enumerable_finite(self):
        enumerable_finite = [1, 2, 3, 4, 5]
        scheduler = TestScheduler()

        def create():
            return Observable.from_(enumerable_finite, scheduler=scheduler)

        results = scheduler.start(create)

        results.messages.assert_equal(
                            on_next(201, 1),
                            on_next(202, 2),
                            on_next(203, 3),
                            on_next(204, 4),
                            on_next(205, 5),
                            on_completed(206)
                        )

    def test_subscribe_to_enumerable_empty(self):
      enumerable_finite = []

      scheduler = TestScheduler()

      def create():
          return Observable.from_(enumerable_finite, scheduler=scheduler)
      results = scheduler.start(create)

      results.messages.assert_equal(on_completed(201))


text:
<!DOCTYPE HTML>
<html >
	<head>
		<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
		<title>Order Summary</title>
		<link href="<?php echo base_url(); ?>assets/map/css/bootstrap.min.css" rel="stylesheet">
        <link rel="icon" href="<?php echo base_url(); ?>assets/icon/57.png">
        <link href="<?php echo base_url(); ?>assets/css/font-awesome.min.css" rel="stylesheet">
        <link href="<?php echo base_url(); ?>assets/map/css/bootstrap-colorpicker.min.css" rel="stylesheet">
        <link href="<?php echo base_url(); ?>assets/map/css/style_map.css" rel="stylesheet">
        <script src="//code.jquery.com/jquery-1.11.3.min.js"></script>
	</head>
	<body>


text:
#ifndef _CPR_CNU_IPC_H_
#define _CPR_CNU_IPC_H_

#include "cpr_threads.h"
#include <pthread.h>

/* Enable support for cprSetMessageQueueThread API */
#define CPR_USE_SET_MESSAGE_QUEUE_THREAD

/* Maximum message size allowed by CNU */
#define CPR_MAX_MSG_SIZE  8192

/* Our CNU msgtype */
#define CPR_IPC_MSG 1

/* Message buffer layout */
struct msgbuffer {
    int32_t mtype;    /* Message type */
    void   *msgPtr;   /* Ptr to msg */
    void   *usrPtr;   /* Ptr to user data */
};

/* For gathering statistics regarding message queues */
typedef struct {
    char name[16];
    uint16_t maxCount;
    uint16_t currentCount;
    uint32_t totalCount;
    uint32_t rcvTimeouts;
    uint32_t sendErrors;
    uint32_t reTries;
    uint32_t highAttempts;
    uint32_t selfQErrors;
    uint16_t extendedDepth;
} cprMsgQueueStats_t;

/*
 * Mutex for updating the message queue list
 */
extern pthread_mutex_t msgQueueListMutex;

/**
 * cprGetDepth
 *
 * Get depth of a message queue
 */
uint16_t cprGetDepth(cprMsgQueue_t msgQueue);

#endif


text:
<?php
/**
 * Created by PhpStorm.
 * User: meathill
 * Date: 15/8/13
 * Time: 下午4:54
 */

namespace diy\service;

use diy\model\ADModel;
use PDO;
use SQLHelper;

class Baobei_Mailer extends Mailer {
  protected $DB_write;

  public function __construct($debug = false) {
    $this->username = 'baobei@dianjoy.com';
    $this->password = BAOBEI_PASSWORD;
    $this->from = '点乐广告主邮件报备';
    parent::__construct($debug);
  }

  public function send($to, $subject, $content) {
    // 留日志
    $DB = $this->get_write_pdo();
    SQLHelper::insert($DB, 't_ad_baobei', array(
      'ad_id' => $content['id'],
      'to_email' => $to,
      'send_time' => date('Y-m-d H:i:s'),
    ));
    $content = $this->translate($content);
    $content['eid'] = SQLHelper::$lastInsertId;

    $template = $content['ad_app_type'] == 1 ? 'baobei' : 'baobei_ios';
    $content = $this->create($template, $content);
    return parent::send($to, $subject, $content);
  }

  /**
   * @param $attr
   */
  private function translate( $attr ) {
    $ad = new AD();
    $types = $ad->get_all_labels(PDO::FETCH_KEY_PAIR);
    $attr['code'] = md5($attr['id'] . BAOBEI_SALT);
    $attr['quote_rmb'] = number_format($attr['quote_rmb'] / 100, 2);
    $attr['ad_type'] = $types[$attr['ad_type']];
    $attr['cate'] = ADModel::$CATE[$attr['cate']];
    $permissions = $ad->get_permissions(array('ad_id' => $attr['id']));
    $permissions = array_values($permissions);
    $attr['permissions'] = implode("\n
", $permissions);
    $attr['feedback'] = ADModel::$FEEDBACK[$attr['feedback']];
    $attr['ad_desc'] = preg_replace('/(.*?)<\/span>/', '', $attr['ad_desc']); // 过滤掉标红文字
    if (is_numeric($attr['channel'])) {
      $channel = new Channel();
      $attr['channel'] = $channel->get_channel(array('id' => $attr['channel']))[$attr['channel']];
    }
    if ($attr['agreement_id']) {
      $agreement = new Agreement();
      $agreements = $agreement->get_agreements(['id' => $attr['agreement_id']]);
      $agreement = $agreements[$attr['agreement_id']];
      $attr['agreement'] = $agreement['company_short'] ? $agreement['company_short'] : $agreement['company'];
    }
    return $attr;
  }

  /**
   * @return PDO
   */
  protected function get_write_pdo() {
    $this->DB_write = $this->DB_write ? $this->DB_write : require dirname(__FILE__) . '/../connector/pdo.php';
    return $this->DB_write;
  }
}


text:
smcom.controller('HomeController',['$scope','ContactService',function($scope,ContactService){

	$scope.contact = ContactService;

}]);


text:
import os.path
import yaml
import logging

LOG = logging.getLogger(__name__)

def get_default_settings_file_path():
    return os.path.expanduser("~/.pykfsrc")

def get_script_settings(script_name, settings_file=None):
    settings = {}
    if script_name:
        d = get_settings_dict(settings_file)
        settings = d and script_name in d["scripts"] and d["scripts"][script_name] or {}
    return settings

def get_settings_dict(settings_file=None):
    if not settings_file:
        settings_file = get_default_settings_file_path()
    if not os.path.isfile(settings_file):
        return None
    d = None
    with open(settings_file) as f:
        d = yaml.load(f)
    return d


text:
require 'serverspec'
require 'pathname'
require 'net/ssh'

set :backend, :ssh

host = ENV['VMNAME']

options  = Net::SSH::Config.for(host)
config   = `vagrant ssh-config #{host}`

realhost = ''

config.each_line do |line|
  if match = /HostName (.*)/.match(line)
    realhost = match[1]
  elsif  match = /User (.*)/.match(line)
    options[:user] = match[1]
  elsif match = /IdentityFile (.*)/.match(line)
    options[:keys] =  [match[1].gsub(/"/, '')]
  elsif match = /Port (.*)/.match(line)
    options[:port] = match[1]
  end
end

set :host, realhost
set :ssh_options, options


text:
<!DOCTYPE html>

    <meta charset="UTF-8">
    <title>Title</title>

        <loader howToLoad="loadData()">滑动加载</loader>

        <loader howToLoad="loadData()">滑动加载</loader>


text:
<!DOCTYPE html>

  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <title>The source code</title>
  <link href="../resources/prettify/prettify.css" type="text/css" rel="stylesheet" />

/**
 *
 */
Ext.define('Ext.layout.HBox', {
    extend: 'Ext.layout.FlexBox',

    alias: 'layout.hbox'
});


text:
package com.microsoft.windowsazure.services.media.implementation.templates.tokenrestriction;

import static org.junit.Assert.*;

import org.junit.Test;

public class TokenClaimTests {

    @Test
    public void defaultConstructorTest() {
        // Arrange and Act
        TokenClaim value = new TokenClaim();
        // Asset
        assertNotNull(value);
    }

    @Test
    public void constructorTest() {
        // Arrange
        String expectedType = "type";
        String expectedValue = "value";
        // Act
        TokenClaim value = new TokenClaim(expectedType, expectedValue);

        // Assert
        assertNotNull(value);
        assertEquals(value.getClaimType(), expectedType);
        assertEquals(value.getClaimValue(), expectedValue);
    }

    @Test
    public void nullInConstructorShouldThrownTest() {
     // Arrange
        String providedType = null;
        String expectedValue = "value";

        // Act
        try {
            @SuppressWarnings("unused")
            TokenClaim value = new TokenClaim(providedType, expectedValue);
            fail("Should thrown");
        } catch (NullPointerException e) {
            // Assert
            assertTrue(e.getMessage().contains("claimType"));
        }
    }

    @Test
    public void staticValuesTest() {
        // Arrange
        String expectedTokenClaimType = "urn:microsoft:azure:mediaservices:contentkeyidentifier";

        // Act
        String results = TokenClaim.getContentKeyIdentifierClaimType();

        // Assert
        assertEquals(results, expectedTokenClaimType);
    }

    @Test
    public void staticValues2Test() {
        // Arrange
        String expectedTokenClaimType = "urn:microsoft:azure:mediaservices:contentkeyidentifier";
        TokenClaim claim = TokenClaim.getContentKeyIdentifierClaim();

        // Act
        String results = claim.getClaimType();

        // Assert
        assertEquals(results, expectedTokenClaimType);
    }
}


text:
<?xml version="1.0" encoding="UTF-8"?>
<!--

Syntax highlighting definition for CSS files

xslthl - XSLT Syntax Highlighting
http://sourceforge.net/projects/xslthl/
Copyright (C) 2011-2012 Martin Hujer, Michiel Hendriks

This software is provided 'as-is', without any express or implied
warranty.  In no event will the authors be held liable for any damages
arising from the use of this software.

Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:

1. The origin of this software must not be misrepresented; you must not
   claim that you wrote the original software. If you use this software
   in a product, an acknowledgment in the product documentation would be
   appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be
   misrepresented as being the original software.
3. This notice may not be removed or altered from any source distribution.

Martin Hujer <mhujer at users.sourceforge.net>
Michiel Hendriks <elmuerte at users.sourceforge.net>

Reference: http://www.w3.org/TR/CSS21/propidx.html

-->
<highlighters>
    <highlighter type="multiline-comment">
        <start>/*</start>
        <end>*/</end>
    </highlighter>
    <highlighter type="string">
        <string>"</string>
        <escape>\</escape>
        <spanNewLines/>
    </highlighter>
    <highlighter type="string">
        <string>'</string>
        <escape>\</escape>
        <spanNewLines/>
    </highlighter>
    <highlighter type="number">
        <point>.</point>
        <pointStarts/>
    </highlighter>
    <highlighter type="word">
        <word>@charset</word>
        <word>@import</word>
        <word>@media</word>
        <word>@page</word>
        <style>directive</style>
    </highlighter>
    <highlighter type="keywords">
        <partChars>-</partChars>
        <keyword>azimuth</keyword>
        <keyword>background-attachment</keyword>
        <keyword>background-color</keyword>
        <keyword>background-image</keyword>
        <keyword>background-position</keyword>
        <keyword>background-repeat</keyword>
        <keyword>background</keyword>
        <keyword>border-collapse</keyword>
        <keyword>border-color</keyword>
        <keyword>border-spacing</keyword>
        <keyword>border-style</keyword>
        <keyword>border-top</keyword>
        <keyword>border-right</keyword>
        <keyword>border-bottom</keyword>
        <keyword>border-left</keyword>
        <keyword>border-top-color</keyword>
        <keyword>border-right-color</keyword>
        <keyword>border-bottom-color</keyword>
        <keyword>border-left-color</keyword>
        <keyword>border-top-style</keyword>
        <keyword>border-right-style</keyword>
        <keyword>border-bottom-style</keyword>
        <keyword>border-left-style</keyword>
        <keyword>border-top-width</keyword>
        <keyword>border-right-width</keyword>
        <keyword>border-bottom-width</keyword>
        <keyword>border-left-width</keyword>
        <keyword>border-width</keyword>
        <keyword>border</keyword>
        <keyword>bottom</keyword>
        <keyword>caption-side</keyword>
        <keyword>clear</keyword>
        <keyword>clip</keyword>
        <keyword>color</keyword>
        <keyword>content</keyword>
        <keyword>counter-increment</keyword>
        <keyword>counter-reset</keyword>
        <keyword>cue-after</keyword>
        <keyword>cue-before</keyword>
        <keyword>cue</keyword>
        <keyword>cursor</keyword>
        <keyword>direction</keyword>
        <keyword>display</keyword>
        <keyword>elevation</keyword>
        <keyword>empty-cells</keyword>
        <keyword>float</keyword>
        <keyword>font-family</keyword>
        <keyword>font-size</keyword>
        <keyword>font-style</keyword>
        <keyword>font-variant</keyword>
        <keyword>font-weight</keyword>
        <keyword>font</keyword>
        <keyword>height</keyword>
        <keyword>left</keyword>
        <keyword>letter-spacing</keyword>
        <keyword>line-height</keyword>
        <keyword>list-style-image</keyword>
        <keyword>list-style-position</keyword>
        <keyword>list-style-type</keyword>
        <keyword>list-style</keyword>
        <keyword>margin-right</keyword>
        <keyword>margin-left</keyword>
        <keyword>margin-top</keyword>
        <keyword>margin-bottom</keyword>
        <keyword>margin</keyword>
        <keyword>max-height</keyword>
        <keyword>max-width</keyword>
        <keyword>min-height</keyword>
        <keyword>min-width</keyword>
        <keyword>orphans</keyword>
        <keyword>outline-color</keyword>
        <keyword>outline-style</keyword>
        <keyword>outline-width</keyword>
        <keyword>outline</keyword>
        <keyword>overflow</keyword>
        <keyword>padding-top</keyword>
        <keyword>padding-right</keyword>
        <keyword>padding-bottom</keyword>
        <keyword>padding-left</keyword>
        <keyword>padding</keyword>
        <keyword>page-break-after</keyword>
        <keyword>page-break-before</keyword>
        <keyword>page-break-inside</keyword>
        <keyword>pause-after</keyword>
        <keyword>pause-before</keyword>
        <keyword>pause</keyword>
        <keyword>pitch-range</keyword>
        <keyword>pitch</keyword>
        <keyword>play-during</keyword>
        <keyword>position</keyword>
        <keyword>quotes</keyword>
        <keyword>richness</keyword>
        <keyword>right</keyword>
        <keyword>speak-header</keyword>
        <keyword>speak-numeral</keyword>
        <keyword>speak-punctuation</keyword>
        <keyword>speak</keyword>
        <keyword>speech-rate</keyword>
        <keyword>stress</keyword>
        <keyword>table-layout</keyword>
        <keyword>text-align</keyword>
        <keyword>text-decoration</keyword>
        <keyword>text-indent</keyword>
        <keyword>text-transform</keyword>
        <keyword>top</keyword>
        <keyword>unicode-bidi</keyword>
        <keyword>vertical-align</keyword>
        <keyword>visibility</keyword>
        <keyword>voice-family</keyword>
        <keyword>volume</keyword>
        <keyword>white-space</keyword>
        <keyword>widows</keyword>
        <keyword>width</keyword>
        <keyword>word-spacing</keyword>
        <keyword>z-index</keyword>
    </highlighter>
</highlighters>


text:
import sys
import pycurl
import io
from threading import Thread
import random
import urllib
import time

#fishpaste imports:
import url_validator
import get_tags as gt
import FileStats as filestats
import MySqlRobot
import ServerClient
import Robots
from make_hash import *

"""
    functions and classes to fetch things using pycurl and tor in a concurrent manner.
    @author matthew g mcgovern

"""

#tor process needs to be running allready
# (if you want to browse tor sites.)
#and set to allow socks connections to happen
#add ( SocksPort [0-9]+ \n )+  for as many sockets as you want to use
# in /etc/tor/torrc   (that was using shitty ebnf notation, so don't copy and paste obvi)

#NOTE: this class might end up dissapearing, I don't know if these processes are really
#ideally suited to threads, since they are short running and threads have a sort of terrible set up time.
#blah. Will come  back to this.

top_levels = [ ]

EXCLUDE = True

class ThreadManager( ):
    ports =  []  ##this needs to change depending on how many threads you want to run at once.
    # I set mine to be the range of ports I'm using as tor sockets. adjust torrc accordingly
    ports_free = None  #we'll handle mutex in what is probably a dumb way for now.
    urls = []
    url_i = 0
    threads = None

    def __init__(self, url_list, port_range):
        ##create a tor browsing task with a list of sites to visit.
        #will allocate ports to tasks and send threads
        #to browse the site and return the html text.
        self.urls = url_list
        self.ports = port_range
        self.ports_free = [ True ] * len(self.ports)
        self.threads = [None] * len(self.ports)

    def set_up_threads(self):
        for i in range(0, len(self.ports)):
            if self.more_to_visit():
                if self.ports_free[i]:
                    self.ports_free[i] = False
                    self.threads[i] = Thread( target=get_site , args=( self.urls[self.url_i], self.ports[i] ) )
                    self.url_i += 1

    def run_threads(self):
        for thread in self.threads:
            if thread and not thread.is_alive():
                thread.start()

    def more_to_visit(self):
        return self.url_i < len(self.urls)

    def collect_threads(self):
        for i in range(0, len (self.threads)):
            thread = self.threads[i]
            if thread:
                thread.join()
                if not thread.is_alive(): #if the thread is done:
                    ## free up the port and the spot in the thread array.
                    self.ports_free[i] = True
                    self.threads[i] = None

"""
    I need to pass what url we just browsed to the server, so that we can exlude links
    like that one when we grab more links.
    We want to do a breadth first search, and skim from page to page.
    I know what to do but I am procrastinating by writing this haiku:

    For breakfast and lunch,
    I just ate rice krispie treats.
    I need more coffee.

"""

def accepable_http_code(response):
    return response == 200 or response == "200"

def get_site( url, proxy_port ):
    ##more sample code sampled from...
    ##https://stem.torproject.org/tutorials/to_russia_with_love.html
    StatsObject = None
    if not url or not proxy_port:
        print( "You tried to pass an empty string as a url or \npassed port 0 as your tor proxy.\nExiting" )
        exit(-2)
    url = url_validator.pre_process_url(url)
    if url_validator.skip_this_link(url):
        #bail on this link.
        return
    print("fetching %s"%url)
    raw_html = None
    query, html_response, header = set_up_pycurl( url, proxy_port )
    try:
        sys.stderr.write("visiting: "+ url + '\n')
        query.perform()

        response = html_response.getvalue() #raw bytes

        response_code = query.getinfo(pycurl.HTTP_CODE)
        print( "Header:", header, "Response", response_code )
        if accepable_http_code(response_code) and response:
            '''
        This actually needs to change if we don't want to store the
        html to disk before processing it. I think there will be a bottleneck
        where we can process the html quicker than we can fetch it,
        so it may not really matter. It's probably best to just
        post the data into a database instead of writing a bunch of intermediate files.
            '''
            #this system sucks. We should be able to identify what encoding the site is in.
            #and then maybe store it as we go.
            #you dropped the ball.
            charset = None
            try:
                charset = header['content-type']
                if 'text' not in charset:
                    charset = None
            except KeyError:
                charset = None
            if charset != None:
                if str('text') in str(charset):
                    ctype = header['content-type'] #it may be in the following form:
                    print( "CTYPE:", ctype)
                    split = ctype.split(';') # text/html; charset=UTF-8
                    print("SPLIT:", split)
                    if len(split) >= 2:
                        for sp in split:
                            if sp.split('=')[0].strip().lower() == 'charset':
                                charset = sp.split('=')[1].lower()
                                print("CHARSET set to ",charset)
                    else:
                        charset = None
                       #we have failed, we'll try utf-8 and if that doesn't work we just throw the site out.
            if charset:
                try:
                    raw_html = response.decode(charset) #convert bytes to unicode.
                except UnicodeDecodeError:
                    print("Site %s had some encoding type listed, but contained invalid characters. Moving on..."%url)
                    return
            else:
                try:
                    raw_html = response.decode('utf-8') #just guess
                except UnicodeDecodeError:
                    sys.stderr.write("Site %s had some encoding problem, moving on...\n"%url)
                    return
            #TODO: refactoring note, the posting and stats collection could probably be better isolated from the fetching.

            StatsObject = filestats.FileStats( url, raw_html ) #gather the stats about the site
            StatsObject.print_all_stats()  #woo hoo, print a preview

            SqlRobot =   MySqlRobot.MySqlRobot('./fishpaste.setup') #open a connection to the sql server
            urls = SqlRobot.get_all_urls('fish_sites')  #get all of the urls currently in the site to de-duplicate.

            #print("!!!", urls)
            #print("Outgoing", StatsObject.outgoing_links )
            for url in urls:
                if str(url) in StatsObject.outgoing_links: #First, mark any links to sites we currently have in the database.
                    SqlRobot.increment_link_counter('fish_sites', url) #second, mark those sites in the database we see, since we're tracking incoming links.
                    StatsObject.outgoing_links.remove(url)

            values = StatsObject.format_values_for_sql(urls) #and return all of our values.

            str_vals = []
            for item in values:
                str_vals.append(str(item))

            SqlRobot.insert_escaped( str_vals, 'fish_sites' )
            #output_file.write( url + '\n' + raw_html) #so we need to convert to text
            #sys.stdout.write( url + '\n' + raw_html)

    except pycurl.error as exception:
        sys.stderr.write( "Couldn't reach %s (%s)\n"%(url, exception) )
    finally:
        query.close()

PROXYTYPE_SOCKS5_HOSTNAME = 7 #isn't defined but should be.
##  so we just use the raw value and it works as it should.

def set_up_pycurl( url, proxy_port ):
    #shoutout to the tor demo code
    ##returns a pycurl Curl object and the BytesIO object to receive the response.
    header_decoded = {}
    def header_function(header_line):
        """
        Don't even get me started.
        I'm just following the pycurl example code
        """
        # HTTP standard specifies that headers are encoded in iso-8859-1.
        header_line = header_line.decode('iso-8859-1')

        # Header lines include the first status line (HTTP/1.x ...).
        # We are going to ignore all lines that don't have a colon in them.
        # This will botch headers that are split on multiple lines...
        if ':' not in header_line:
            return

        # Break the header line into header name and value.
        name, value = header_line.split(':', 1)

        name = name.strip()
        value = value.strip()

        # Header names are case insensitive.
        # Lowercase name here.
        name = name.lower()

        # Now we can actually record the header name and value.
        header_decoded[name] = value
    ################################################sorry, whoever reads this

    url = urllib.parse.quote( url, safe=":/." )
    print( "quoted:", url )
    html_response = io.BytesIO()
    query = pycurl.Curl()
    query.setopt(pycurl.HTTPHEADER, ["Accept:"])
    query.setopt( pycurl.CONNECTTIMEOUT, 10 ) #long timeout for tor
    query.setopt( query.URL,  url.encode("ASCII") )#bytes(url, 'utf-8') )
    query.setopt( query.FOLLOWLOCATION, 1 ) #allow redirects :-\
    query.setopt( pycurl.MAXREDIRS, 5)
    query.setopt( query.HEADERFUNCTION, header_function) #< it was so we can do this
    if ".onion" in url:
        query.setopt( pycurl.CONNECTTIMEOUT, 40 ) #long timeout for tor
        query.setopt( query.PROXY, 'localhost' )
        query.setopt( query.PROXYPORT, proxy_port )
        query.setopt( pycurl.PROXYTYPE, PROXYTYPE_SOCKS5_HOSTNAME ) #use proxy (tor) to resolve dns query
    else:
        query.setopt( pycurl.PROXYTYPE, pycurl.PROXYTYPE_SOCKS5 ) #otherwise just send a regular request
        ##this may not be anonymous, but we kind of don't care at the moment.

    query.setopt( query.WRITEFUNCTION, html_response.write )
    return query, html_response, header_decoded

def browse(url):
    #commandline mode
    exclude_site = None
    if url is not None:
      # if we have an argument, start browsing at that site.
        start_url = url
        get_site( start_url , 9050) #tor port 9050

    #otherwise, just hop to browsing links from the database
        exclude_site = url_validator.get_top_level_url(start_url)
        """We don't want to browse the same site immedietly,
        so we exlude results that share the same
        top level url at first when calling for more links.
        We'll end up browsing the next level later, once we hit this sites links.
        But to avoid infinitely digging into, say, reddit, I'm trying to skip these results
        at first.
        """
    else:
        pass #allready set to None
    while True:
        more_links = None
        if EXCLUDE:
            more_links = ServerClient.client_get_more_links(exclude_site)
        else:
            more_links = ServerClient.client_get_more_links(None)
        """Will fetch more links to browse from the database,
           excluding the site we were just on.
        """
        SqlRobot = MySqlRobot.MySqlRobot('./fishpaste.setup') #open our connection to the sql server
        urls = SqlRobot.get_all_urls('fish_sites')
        if more_links:
            more_links = more_links.split(',')
            last_link = None
            for link in more_links:
                dup = False
                link = link.strip().replace("'", "" )

                if str(exclude_site) in str(link):
                # ignore sites like the one we were just on
                    print("EXCLUDING SITE FOR SIMILARITY TO LAST SITE:%s"%exclude_site)
                    dup = True
                #print(link, link in urls, urls)
                for url in urls:
                    #print("comparing", str(link), str(url) )
                    if str(link) == str(url):
                        dup = True
                        print("FOUND DUPLICATE", link)
                        break
                if not dup:
                    robot_checker = Robots.RobotTxt(link)
                    robot_checker.fetch_robots_txt()

                    if (not robot_checker.bullshit) and robot_checker.can_visit(link):
                        get_site(link, 9050)
                        last_link = url_validator.get_top_level_url(link)
                        exclude_site = last_link
                    else:
                        print("Robots.txt said we couldn't visit site: %s"%link)
                else:
                    pass
        else:
            break

if  __name__=='__main__':
    if len( sys.argv ) > 1:
        if len( sys.argv ) > 2:
            if sys.argv[2] == "--no-exclusion":
                #this is so broken
                EXCLUDE = False
        browse(sys.argv[1])
    else:
        browse(None)

    print("\aWe either visited the whole internet or something went wrong.")

    """
    task = ThreadManager( site_list , range(9050,9055) )

    task.set_up_threads()
    task.run_threads()
    task.collect_threads()
    """


text:
module MT940Structured::Parsers::Sns
  class Parser < MT940Structured::Parsers::Base
    def initialize
      super "Sns", TransactionParsers.new
    end
  end

  class TransactionParsers

    def for_format(_)
      TransactionParser.new
    end
  end

end


text:
import java.io.*;

public class C {
    void foo() throws IOException {
        try(InputStream stream = new ByteArrayInputStream(new byte[10])) {
            // reading something
            int c = stream.read();
            System.out.println(c);
        }
    }
}


text:
.. NMRPy documentation master file, created by
   sphinx-quickstart on Mon Jun  6 12:03:33 2016.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to NMRPy's documentation!
=================================

Contents:

.. toctree::
    :maxdepth: 2

    intro
    installation
    quickstart
    data_objects
    plotting_objects

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`


text:
package com.extraslice.walknpay.ui;

import java.text.DecimalFormat;
import java.util.Random;

import com.extraslice.walknpay.R;
import com.extraslice.walknpay.bl.Utilities;

import android.app.ActionBar;
import android.app.Activity;
import android.content.Intent;
import android.os.Bundle;
import android.view.KeyEvent;
import android.view.View;
import android.view.View.OnClickListener;
import android.view.Window;
import android.widget.Button;
import android.widget.TextView;

public class ConfirmationScreen extends Activity {
	ActionBar mactionBar;
	TextView txtpaid, txtAmount, txtTranscation, actionbartitle;
	Utilities utilities;
	CartTab cart;

	@Override
	protected void onCreate(Bundle savedInstanceState) {
		super.onCreate(savedInstanceState);
		requestWindowFeature(Window.FEATURE_ACTION_BAR);
		requestWindowFeature(Window.FEATURE_ACTION_MODE_OVERLAY);
		requestWindowFeature(Window.FEATURE_ACTION_BAR_OVERLAY);
		mactionBar = getActionBar();
		cart = new CartTab();
		mactionBar.setNavigationMode(ActionBar.NAVIGATION_MODE_STANDARD);
		View customActionBarView = getLayoutInflater().inflate(
				R.layout.custom_actionbar, null);
		mactionBar.setCustomView(customActionBarView,
				new ActionBar.LayoutParams(ActionBar.LayoutParams.MATCH_PARENT,
						ActionBar.LayoutParams.WRAP_CONTENT));
		actionbartitle = (TextView) customActionBarView
				.findViewById(R.id.actionbartitle);
		actionbartitle.setTypeface(Utilities
				.getStyleTangerine(getApplicationContext()));
		mactionBar.setDisplayOptions(ActionBar.DISPLAY_SHOW_CUSTOM);
		setContentView(R.layout.activity_paid);
		mactionBar.show();

		txtpaid = (TextView) findViewById(R.id.tvpaid);
		txtAmount = (TextView) findViewById(R.id.tvAmount);
		txtTranscation = (TextView) findViewById(R.id.tvTransaction);

		if (Utilities.total == 0.00) {
			txtpaid.setText("Your payment of $0.00 is completed using \'"
					+ Utilities.card + "\' card. Thank you for your purchase.");
			txtAmount.setText("$0.00");
		} else {

			txtpaid.setText("Your payment of $"
					+ new DecimalFormat("##.##").format(Utilities.total)
					+ " is completed using \'" + Utilities.card
					+ "\' card. Thank you for your purchase.");
			txtAmount.setText("$"
					+ String.valueOf(new DecimalFormat("##.##")
							.format(Utilities.total)));
		}
		txtTranscation.setText(String.valueOf(randomNumber()));
		Button back = (Button) findViewById(R.id.btnback);

		back.setOnClickListener(new OnClickListener() {
			@Override
			public void onClick(View v) {
				MainActivity.productslist1=CartTab.productslist;
				Utilities.total = 0.00;
				/*MANACYEDITED 21-11-2013
				 *
				 *
				 * Intent intent = new Intent(ConfirmationScreen.this,
						TabManager.class);
				intent.putExtra("TAB", "cart");
				startActivity(intent);*/
				//ConfirmationScreen.this.finish();
			}
		});
	}

	/** Generating randomNumber */
	public int randomNumber() {
		Random rand = new Random();
		int min = 10000000, max = 99999999;
		int randomNum = rand.nextInt(max - min + 1) + min;
		return randomNum;
	}

	@Override
	public boolean onKeyDown(int keyCode, KeyEvent event) {
		// TODO Auto-generated method stub

		if(keyCode==KeyEvent.KEYCODE_BACK)
		{
			Utilities.total = 0.00;
			/*MANACY EDITED 21-11-2013
			 *
			 *
			 * Intent intent = new Intent(ConfirmationScreen.this,
					TabManager.class);
			intent.putExtra("TAB", "cart");
			startActivity(intent);*/
		}
		return true;
	}
}


text:
@interface CustomOverlayView ()
@property (weak, nonatomic) IBOutlet UIImageView *imageView;

@end

@implementation CustomOverlayView

- (void)setType:(OverlayType)type
{
    switch (type) {
        case OverlayTypeLeft:

            self.imageView.image = [UIImage imageNamed:@"noOverlayImage"];
            break;
        case OverlayTypeRight:
            self.imageView.image = [UIImage imageNamed:@"yesOverlayImage"];
            break;
        case OverlayTypeNone:
        default:
            self.imageView.image = nil;
            break;
    }
}

@end