text:
.. _module-pw_tokenizer:

============
pw_tokenizer
============
:bdg-primary:`host`
:bdg-primary:`device`
:bdg-secondary:`Python`
:bdg-secondary:`C++`
:bdg-secondary:`TypeScript`
:bdg-success:`stable`

Logging is critical, but developers are often forced to choose between
additional logging or saving crucial flash space. The ``pw_tokenizer`` module
helps address this by replacing printf-style strings with binary tokens during
compilation. This enables extensive logging with substantially less memory
usage.

.. note::
  This usage of the term "tokenizer" is not related to parsing! The
  module is called tokenizer because it replaces a whole string literal with an
  integer token. It does not parse strings into separate tokens.

The most common application of ``pw_tokenizer`` is binary logging, and it is
designed to integrate easily into existing logging systems. However, the
tokenizer is general purpose and can be used to tokenize any strings, with or
without printf-style arguments.

**Why tokenize strings?**

* Dramatically reduce binary size by removing string literals from binaries.
* Reduce I/O traffic, RAM, and flash usage by sending and storing compact tokens
  instead of strings. We've seen over 50% reduction in encoded log contents.
* Reduce CPU usage by replacing snprintf calls with simple tokenization code.
* Remove potentially sensitive log, assert, and other strings from binaries.

--------------
Basic overview
--------------
There are two sides to ``pw_tokenizer``, which we call tokenization and
detokenization.

* **Tokenization** converts string literals in the source code to binary tokens
  at compile time. If the string has printf-style arguments, these are encoded
  to compact binary form at runtime.
* **Detokenization** converts tokenized strings back to the original
  human-readable strings.

Here's an overview of what happens when ``pw_tokenizer`` is used:

1. During compilation, the ``pw_tokenizer`` module hashes string literals to
   generate stable 32-bit tokens.
2. The tokenization macro removes these strings by declaring them in an ELF
   section that is excluded from the final binary.
3. After compilation, strings are extracted from the ELF to build a database of
   tokenized strings for use by the detokenizer. The ELF file may also be used
   directly.
4. During operation, the device encodes the string token and its arguments, if
   any.
5. The encoded tokenized strings are sent off-device or stored.
6. Off-device, the detokenizer tools use the token database to decode the
   strings to human-readable form.

Example: tokenized logging
==========================
This example demonstrates using ``pw_tokenizer`` for logging. In this example,
tokenized logging saves ~90% in binary size (41 → 4 bytes) and 70% in encoded
size (49 → 15 bytes).

**Before**: plain text logging

+------------------+-------------------------------------------+---------------+
| Location         | Logging Content                           | Size in bytes |
+==================+===========================================+===============+
| Source contains  | ``LOG("Battery state: %s; battery         |               |
|                  | voltage: %d mV", state, voltage);``       |               |
+------------------+-------------------------------------------+---------------+
| Binary contains  | ``"Battery state: %s; battery             | 41            |
|                  | voltage: %d mV"``                         |               |
+------------------+-------------------------------------------+---------------+
|                  | (log statement is called with             |               |
|                  | ``"CHARGING"`` and ``3989`` as arguments) |               |
+------------------+-------------------------------------------+---------------+
| Device transmits | ``"Battery state: CHARGING; battery       | 49            |
|                  | voltage: 3989 mV"``                       |               |
+------------------+-------------------------------------------+---------------+
| When viewed      | ``"Battery state: CHARGING; battery       |               |
|                  | voltage: 3989 mV"``                       |               |
+------------------+-------------------------------------------+---------------+

**After**: tokenized logging

+------------------+-----------------------------------------------------------+---------+
| Location         | Logging Content                                           | Size in |
|                  |                                                           | bytes   |
+==================+===========================================================+=========+
| Source contains  | ``LOG("Battery state: %s; battery                         |         |
|                  | voltage: %d mV", state, voltage);``                       |         |
+------------------+-----------------------------------------------------------+---------+
| Binary contains  | ``d9 28 47 8e`` (0x8e4728d9)                              | 4       |
+------------------+-----------------------------------------------------------+---------+
|                  | (log statement is called with                             |         |
|                  | ``"CHARGING"`` and ``3989`` as arguments)                 |         |
+------------------+-----------------------------------------------------------+---------+
| Device transmits | =============== ============================== ========== | 15      |
|                  | ``d9 28 47 8e`` ``08 43 48 41 52 47 49 4E 47`` ``aa 3e``  |         |
|                  | --------------- ------------------------------ ---------- |         |
|                  | Token           ``"CHARGING"`` argument        ``3989``,  |         |
|                  |                                                as         |         |
|                  |                                                varint     |         |
|                  | =============== ============================== ========== |         |
+------------------+-----------------------------------------------------------+---------+
| When viewed      | ``"Battery state: CHARGING; battery voltage: 3989 mV"``   |         |
+------------------+-----------------------------------------------------------+---------+

---------------
Getting started
---------------
Integrating ``pw_tokenizer`` requires a few steps beyond building the code. This
section describes one way ``pw_tokenizer`` might be integrated with a project.
These steps can be adapted as needed.

1. Add ``pw_tokenizer`` to your build. Build files for GN, CMake, and Bazel are
   provided. For Make or other build systems, add the files specified in the
   BUILD.gn's ``pw_tokenizer`` target to the build.
2. Use the tokenization macros in your code. See `Tokenization`_.
3. Add the contents of ``pw_tokenizer_linker_sections.ld`` to your project's
   linker script. In GN and CMake, this step is done automatically.
4. Compile your code to produce an ELF file.
5. Run ``database.py create`` on the ELF file to generate a CSV token
   database. See `Managing token databases`_.
6. Commit the token database to your repository. See notes in `Database
   management`_.
7. Integrate a ``database.py add`` command to your build to automatically update
   the committed token database. In GN, use the ``pw_tokenizer_database``
   template to do this. See `Update a database`_.
8. Integrate ``detokenize.py`` or the C++ detokenization library with your tools
   to decode tokenized logs. See `Detokenization`_.

Using with Zephyr
=================
When building ``pw_tokenizer`` with Zephyr, 3 Kconfigs can be used currently:

* ``CONFIG_PIGWEED_TOKENIZER`` will automatically link ``pw_tokenizer`` as well
  as any dependencies.
* ``CONFIG_PIGWEED_TOKENIZER_BASE64`` will automatically link
  ``pw_tokenizer.base64`` as well as any dependencies.
* ``CONFIG_PIGWEED_DETOKENIZER`` will automatically link
  ``pw_tokenizer.decoder`` as well as any dependencies.

Once enabled, the tokenizer headers can be included like any Zephyr headers:

.. code-block:: cpp

   #include <pw_tokenizer/tokenize.h>

------------
Tokenization
------------
Tokenization converts a string literal to a token. If it's a printf-style
string, its arguments are encoded along with it. The results of tokenization can
be sent off device or stored in place of a full string.

Tokenization macros
===================
Adding tokenization to a project is simple. To tokenize a string, include
``pw_tokenizer/tokenize.h`` and invoke one of the ``PW_TOKENIZE_`` macros.

Tokenize a string literal
-------------------------
The ``PW_TOKENIZE_STRING`` macro converts a string literal to a ``uint32_t``
token.

.. code-block:: cpp

   constexpr uint32_t token = PW_TOKENIZE_STRING("Any string literal!");

.. admonition:: When to use this macro

   Use ``PW_TOKENIZE_STRING`` to tokenize string literals that do not have
   %-style arguments.

Tokenize to a handler function
------------------------------
``PW_TOKENIZE_TO_GLOBAL_HANDLER`` is the most efficient tokenization function,
since it takes the fewest arguments. It encodes a tokenized string to a
buffer on the stack. The size of the buffer is set with
``PW_TOKENIZER_CFG_ENCODING_BUFFER_SIZE_BYTES``.

This macro is provided by the ``pw_tokenizer:global_handler`` facade. The
backend for this facade must define the ``pw_tokenizer_HandleEncodedMessage``
C-linkage function.

.. code-block:: cpp

   PW_TOKENIZE_TO_GLOBAL_HANDLER(format_string_literal, arguments...);

   void pw_tokenizer_HandleEncodedMessage(const uint8_t encoded_message[],
                                          size_t size_bytes);

``PW_TOKENIZE_TO_GLOBAL_HANDLER_WITH_PAYLOAD`` is similar, but passes a
``uintptr_t`` argument to the global handler function. Values like a log level
can be packed into the ``uintptr_t``.

This macro is provided by the ``pw_tokenizer:global_handler_with_payload``
facade. The backend for this facade must define the
``pw_tokenizer_HandleEncodedMessageWithPayload`` C-linkage function.

.. code-block:: cpp

   PW_TOKENIZE_TO_GLOBAL_HANDLER_WITH_PAYLOAD(payload,
                                              format_string_literal,
                                              arguments...);

   void pw_tokenizer_HandleEncodedMessageWithPayload(
       uintptr_t payload, const uint8_t encoded_message[], size_t size_bytes);

.. admonition:: When to use these macros

   Use anytime a global handler is sufficient, particularly for widely expanded
   macros, like a logging macro. ``PW_TOKENIZE_TO_GLOBAL_HANDLER`` or
   ``PW_TOKENIZE_TO_GLOBAL_HANDLER_WITH_PAYLOAD`` are the most efficient macros
   for tokenizing printf-style strings.

Tokenize to a callback
----------------------
``PW_TOKENIZE_TO_CALLBACK`` tokenizes to a buffer on the stack and calls a
``void(const uint8_t* buffer, size_t buffer_size)`` callback that is provided at
the call site. The size of the buffer is set with
``PW_TOKENIZER_CFG_ENCODING_BUFFER_SIZE_BYTES``.

.. code-block:: cpp

   PW_TOKENIZE_TO_CALLBACK(HandlerFunction, "Format string: %x", arguments...);

.. admonition:: When to use this macro

   Use ``PW_TOKENIZE_TO_CALLBACK`` if the global handler version is already in
   use for another purpose or more flexibility is needed.

Tokenize to a buffer
--------------------
The most flexible tokenization macro is ``PW_TOKENIZE_TO_BUFFER``, which encodes
to a caller-provided buffer.

.. code-block:: cpp

   uint8_t buffer[BUFFER_SIZE];
   size_t size_bytes = sizeof(buffer);
   PW_TOKENIZE_TO_BUFFER(buffer, &size_bytes, format_string_literal, arguments...);

While ``PW_TOKENIZE_TO_BUFFER`` is maximally flexible, it takes more arguments
than the other macros, so its per-use code size overhead is larger.

.. admonition:: When to use this macro

   Use ``PW_TOKENIZE_TO_BUFFER`` to encode to a custom-sized buffer or if the
   other macros are insufficient. Avoid using ``PW_TOKENIZE_TO_BUFFER`` in
   widely expanded macros, such as a logging macro, because it will result in
   larger code size than its alternatives.

.. _module-pw_tokenizer-custom-macro:

Tokenize with a custom macro
----------------------------
Projects may need more flexbility than the standard ``pw_tokenizer`` macros
provide. To support this, projects may define custom tokenization macros. This
requires the use of two low-level ``pw_tokenizer`` macros:

.. c:macro:: PW_TOKENIZE_FORMAT_STRING(domain, mask, format, ...)

   Tokenizes a format string and sets the ``_pw_tokenizer_token`` variable to the
   token. Must be used in its own scope, since the same variable is used in every
   invocation.

   The tokenized string uses the specified :ref:`tokenization domain
   <module-pw_tokenizer-domains>`.  Use ``PW_TOKENIZER_DEFAULT_DOMAIN`` for the
   default. The token also may be masked; use ``UINT32_MAX`` to keep all bits.

.. c:macro:: PW_TOKENIZER_ARG_TYPES(...)

   Converts a series of arguments to a compact format that replaces the format
   string literal.

Use these two macros within the custom tokenization macro to call a function
that does the encoding. The following example implements a custom tokenization
macro for use with :ref:`module-pw_log_tokenized`.

.. code-block:: cpp

   #include "pw_tokenizer/tokenize.h"

   #ifndef __cplusplus
   extern "C" {
   #endif

   void EncodeTokenizedMessage(pw_tokenizer_Payload metadata,
                               pw_tokenizer_Token token,
                               pw_tokenizer_ArgTypes types,
                               ...);

   #ifndef __cplusplus
   }  // extern "C"
   #endif

   #define PW_LOG_TOKENIZED_ENCODE_MESSAGE(metadata, format, ...)         \
     do {                                                                 \
       PW_TOKENIZE_FORMAT_STRING(                                         \
           PW_TOKENIZER_DEFAULT_DOMAIN, UINT32_MAX, format, __VA_ARGS__); \
       EncodeTokenizedMessage(payload,                                    \
                              _pw_tokenizer_token,                        \
                              PW_TOKENIZER_ARG_TYPES(__VA_ARGS__)         \
                                  PW_COMMA_ARGS(__VA_ARGS__));            \
     } while (0)

In this example, the ``EncodeTokenizedMessage`` function would handle encoding
and processing the message. Encoding is done by the
``pw::tokenizer::EncodedMessage`` class or ``pw::tokenizer::EncodeArgs``
function from ``pw_tokenizer/encode_args.h``. The encoded message can then be
transmitted or stored as needed.

.. code-block:: cpp

   #include "pw_log_tokenized/log_tokenized.h"
   #include "pw_tokenizer/encode_args.h"

   void HandleTokenizedMessage(pw::log_tokenized::Metadata metadata,
                               pw::span<std::byte> message);

   extern "C" void EncodeTokenizedMessage(const pw_tokenizer_Payload metadata,
                                          const pw_tokenizer_Token token,
                                          const pw_tokenizer_ArgTypes types,
                                          ...) {
     va_list args;
     va_start(args, types);
     pw::tokenizer::EncodedMessage encoded_message(token, types, args);
     va_end(args);

     HandleTokenizedMessage(metadata, encoded_message);
   }

.. admonition:: When to use a custom macro

   Use existing tokenization macros whenever possible. A custom macro may be
   needed to support use cases like the following:

   * Variations of ``PW_TOKENIZE_TO_GLOBAL_HANDLER_WITH_PAYLOAD`` that take
     different arguments.
   * Supporting global handler macros that use different handler functions.

Binary logging with pw_tokenizer
================================
String tokenization is perfect for logging. Consider the following log macro,
which gathers the file, line number, and log message. It calls the ``RecordLog``
function, which formats the log string, collects a timestamp, and transmits the
result.

.. code-block:: cpp

   #define LOG_INFO(format, ...) \
       RecordLog(LogLevel_INFO, __FILE_NAME__, __LINE__, format, ##__VA_ARGS__)

   void RecordLog(LogLevel level, const char* file, int line, const char* format,
                  ...) {
     if (level < current_log_level) {
       return;
     }

     int bytes = snprintf(buffer, sizeof(buffer), "%s:%d ", file, line);

     va_list args;
     va_start(args, format);
     bytes += vsnprintf(&buffer[bytes], sizeof(buffer) - bytes, format, args);
     va_end(args);

     TransmitLog(TimeSinceBootMillis(), buffer, size);
   }

It is trivial to convert this to a binary log using the tokenizer. The
``RecordLog`` call is replaced with a
``PW_TOKENIZE_TO_GLOBAL_HANDLER_WITH_PAYLOAD`` invocation. The
``pw_tokenizer_HandleEncodedMessageWithPayload`` implementation collects the
timestamp and transmits the message with ``TransmitLog``.

.. code-block:: cpp

   #define LOG_INFO(format, ...)                   \
       PW_TOKENIZE_TO_GLOBAL_HANDLER_WITH_PAYLOAD( \
           (pw_tokenizer_Payload)LogLevel_INFO,    \
           __FILE_NAME__ ":%d " format,            \
           __LINE__,                               \
           __VA_ARGS__);                           \

   extern "C" void pw_tokenizer_HandleEncodedMessageWithPayload(
       uintptr_t level, const uint8_t encoded_message[], size_t size_bytes) {
     if (static_cast<LogLevel>(level) >= current_log_level) {
       TransmitLog(TimeSinceBootMillis(), encoded_message, size_bytes);
     }
   }

Note that the ``__FILE_NAME__`` string is directly included in the log format
string. Since the string is tokenized, this has no effect on binary size. A
``%d`` for the line number is added to the format string, so that changing the
line of the log message does not generate a new token. There is no overhead for
additional tokens, but it may not be desirable to fill a token database with
duplicate log lines.

Tokenizing function names
=========================
The string literal tokenization functions support tokenizing string literals or
constexpr character arrays (``constexpr const char[]``). In GCC and Clang, the
special ``__func__`` variable and ``__PRETTY_FUNCTION__`` extension are declared
as ``static constexpr char[]`` in C++ instead of the standard ``static const
char[]``. This means that ``__func__`` and ``__PRETTY_FUNCTION__`` can be
tokenized while compiling C++ with GCC or Clang.

.. code-block:: cpp

   // Tokenize the special function name variables.
   constexpr uint32_t function = PW_TOKENIZE_STRING(__func__);
   constexpr uint32_t pretty_function = PW_TOKENIZE_STRING(__PRETTY_FUNCTION__);

   // Tokenize the function name variables to a handler function.
   PW_TOKENIZE_TO_GLOBAL_HANDLER(__func__)
   PW_TOKENIZE_TO_GLOBAL_HANDLER(__PRETTY_FUNCTION__)

Note that ``__func__`` and ``__PRETTY_FUNCTION__`` are not string literals.
They are defined as static character arrays, so they cannot be implicitly
concatentated with string literals. For example, ``printf(__func__ ": %d",
123);`` will not compile.

Tokenization in Python
======================
The Python ``pw_tokenizer.encode`` module has limited support for encoding
tokenized messages with the ``encode_token_and_args`` function.

.. autofunction:: pw_tokenizer.encode.encode_token_and_args

This function requires a string's token is already calculated. Typically these
tokens are provided by a database, but they can be manually created using the
tokenizer hash.

.. autofunction:: pw_tokenizer.tokens.pw_tokenizer_65599_hash

This is particularly useful for offline token database generation in cases where
tokenized strings in a binary cannot be embedded as parsable pw_tokenizer
entries.

.. note::
   In C, the hash length of a string has a fixed limit controlled by
   ``PW_TOKENIZER_CFG_C_HASH_LENGTH``. To match tokens produced by C (as opposed
   to C++) code, ``pw_tokenizer_65599_hash()`` should be called with a matching
   hash length limit. When creating an offline database, it's a good idea to
   generate tokens for both, and merge the databases.

Encoding
========
The token is a 32-bit hash calculated during compilation. The string is encoded
little-endian with the token followed by arguments, if any. For example, the
31-byte string ``You can go about your business.`` hashes to 0xdac9a244.
This is encoded as 4 bytes: ``44 a2 c9 da``.

Arguments are encoded as follows:

* **Integers**  (1--10 bytes) --
  `ZagZag and varint encoded <https://developers.google.com/protocol-buffers/docs/encoding#signed-integers>`_,
  similarly to Protocol Buffers. Smaller values take fewer bytes.
* **Floating point numbers** (4 bytes) -- Single precision floating point.
* **Strings** (1--128 bytes) -- Length byte followed by the string contents.
  The top bit of the length whether the string was truncated or not. The
  remaining 7 bits encode the string length, with a maximum of 127 bytes.

.. TODO(hepler): insert diagram here!

.. tip::
   ``%s`` arguments can quickly fill a tokenization buffer. Keep ``%s``
   arguments short or avoid encoding them as strings (e.g. encode an enum as an
   integer instead of a string). See also `Tokenized strings as %s arguments`_.

Encoding command line utility
-----------------------------
The ``pw_tokenizer.encode`` command line tool can be used to encode tokenized
strings.

.. code-block:: bash

  python -m pw_tokenizer.encode [-h] FORMAT_STRING [ARG ...]

Example:

.. code-block:: text

  $ python -m pw_tokenizer.encode "There's... %d many of %s!" 2 them
        Raw input: "There's... %d many of %s!" % (2, 'them')
  Formatted input: There's... 2 many of them!
            Token: 0xb6ef8b2d
          Encoded: b'-\x8b\xef\xb6\x04\x04them' (2d 8b ef b6 04 04 74 68 65 6d) [10 bytes]
  Prefixed Base64: $LYvvtgQEdGhlbQ==

See ``--help`` for full usage details.

Token generation: fixed length hashing at compile time
======================================================
String tokens are generated using a modified version of the x65599 hash used by
the SDBM project. All hashing is done at compile time.

In C code, strings are hashed with a preprocessor macro. For compatibility with
macros, the hash must be limited to a fixed maximum number of characters. This
value is set by ``PW_TOKENIZER_CFG_C_HASH_LENGTH``. Increasing
``PW_TOKENIZER_CFG_C_HASH_LENGTH`` increases the compilation time for C due to
the complexity of the hashing macros.

C++ macros use a constexpr function instead of a macro. This function works with
any length of string and has lower compilation time impact than the C macros.
For consistency, C++ tokenization uses the same hash algorithm, but the
calculated values will differ between C and C++ for strings longer than
``PW_TOKENIZER_CFG_C_HASH_LENGTH`` characters.

.. _module-pw_tokenizer-domains:

Tokenization domains
====================
``pw_tokenizer`` supports having multiple tokenization domains. Domains are a
string label associated with each tokenized string. This allows projects to keep
tokens from different sources separate. Potential use cases include the
following:

* Keep large sets of tokenized strings separate to avoid collisions.
* Create a separate database for a small number of strings that use truncated
  tokens, for example only 10 or 16 bits instead of the full 32 bits.

If no domain is specified, the domain is empty (``""``). For many projects, this
default domain is sufficient, so no additional configuration is required.

.. code-block:: cpp

   // Tokenizes this string to the default ("") domain.
   PW_TOKENIZE_STRING("Hello, world!");

   // Tokenizes this string to the "my_custom_domain" domain.
   PW_TOKENIZE_STRING_DOMAIN("my_custom_domain", "Hello, world!");

The database and detokenization command line tools default to reading from the
default domain. The domain may be specified for ELF files by appending
``#DOMAIN_NAME`` to the file path. Use ``#.*`` to read from all domains. For
example, the following reads strings in ``some_domain`` from ``my_image.elf``.

.. code-block:: sh

   ./database.py create --database my_db.csv path/to/my_image.elf#some_domain

See `Managing token databases`_ for information about the ``database.py``
command line tool.

Smaller tokens with masking
===========================
``pw_tokenizer`` uses 32-bit tokens. On 32-bit or 64-bit architectures, using
fewer than 32 bits does not improve runtime or code size efficiency. However,
when tokens are packed into data structures or stored in arrays, the size of the
token directly affects memory usage. In those cases, every bit counts, and it
may be desireable to use fewer bits for the token.

``pw_tokenizer`` allows users to provide a mask to apply to the token. This
masked token is used in both the token database and the code. The masked token
is not a masked version of the full 32-bit token, the masked token is the token.
This makes it trivial to decode tokens that use fewer than 32 bits.

Masking functionality is provided through the ``*_MASK`` versions of the macros.
For example, the following generates 16-bit tokens and packs them into an
existing value.

.. code-block:: cpp

   constexpr uint32_t token = PW_TOKENIZE_STRING_MASK("domain", 0xFFFF, "Pigweed!");
   uint32_t packed_word = (other_bits << 16) | token;

Tokens are hashes, so tokens of any size have a collision risk. The fewer bits
used for tokens, the more likely two strings are to hash to the same token. See
`token collisions`_.

Masked tokens without arguments may be encoded in fewer bytes. For example, the
16-bit token ``0x1234`` may be encoded as two little-endian bytes (``34 12``)
rather than four (``34 12 00 00``). The detokenizer tools zero-pad data smaller
than four bytes. Tokens with arguments must always be encoded as four bytes.

Token collisions
================
Tokens are calculated with a hash function. It is possible for different
strings to hash to the same token. When this happens, multiple strings will have
the same token in the database, and it may not be possible to unambiguously
decode a token.

The detokenization tools attempt to resolve collisions automatically. Collisions
are resolved based on two things:

- whether the tokenized data matches the strings arguments' (if any), and
- if / when the string was marked as having been removed from the database.

Working with collisions
-----------------------
Collisions may occur occasionally. Run the command
``python -m pw_tokenizer.database report <database>`` to see information about a
token database, including any collisions.

If there are collisions, take the following steps to resolve them.

- Change one of the colliding strings slightly to give it a new token.
- In C (not C++), artificial collisions may occur if strings longer than
  ``PW_TOKENIZER_CFG_C_HASH_LENGTH`` are hashed. If this is happening, consider
  setting ``PW_TOKENIZER_CFG_C_HASH_LENGTH`` to a larger value.  See
  ``pw_tokenizer/public/pw_tokenizer/config.h``.
- Run the ``mark_removed`` command with the latest version of the build
  artifacts to mark missing strings as removed. This deprioritizes them in
  collision resolution.

  .. code-block:: sh

     python -m pw_tokenizer.database mark_removed --database <database> <ELF files>

  The ``purge`` command may be used to delete these tokens from the database.

Probability of collisions
-------------------------
Hashes of any size have a collision risk. The probability of one at least
one collision occurring for a given number of strings is unintuitively high
(this is known as the `birthday problem
<https://en.wikipedia.org/wiki/Birthday_problem>`_). If fewer than 32 bits are
used for tokens, the probability of collisions increases substantially.

This table shows the approximate number of strings that can be hashed to have a
1% or 50% probability of at least one collision (assuming a uniform, random
hash).

+-------+---------------------------------------+
| Token | Collision probability by string count |
| bits  +--------------------+------------------+
|       |         50%        |          1%      |
+=======+====================+==================+
|   32  |       77000        |        9300      |
+-------+--------------------+------------------+
|   31  |       54000        |        6600      |
+-------+--------------------+------------------+
|   24  |        4800        |         580      |
+-------+--------------------+------------------+
|   16  |         300        |          36      |
+-------+--------------------+------------------+
|    8  |          19        |           3      |
+-------+--------------------+------------------+

Keep this table in mind when masking tokens (see `Smaller tokens with
masking`_). 16 bits might be acceptable when tokenizing a small set of strings,
such as module names, but won't be suitable for large sets of strings, like log
messages.

Using tokenization in expressions
=================================
The tokenization macros above cannot be used inside expressions. For example,
the following code snippet will fail to compile:

.. code-block:: cpp

   DoSomething(PW_TOKENIZE_STRING("Fail"));

An alternate set of macros are provided for use inside expressions. These make
use of lambda functions, so while they can be used inside expressions, they
cannot be assigned to constexpr variables or be used with special function
variables like ``__func__``.

The following tokenization macros may be used inside epxressions:

* ``PW_TOKENIZE_STRING_EXPR``
* ``PW_TOKENIZE_STRING_DOMAIN_EXPR``
* ``PW_TOKENIZE_STRING_MASK_EXPR``

For example, the following code snippet will work:

.. code-block:: cpp

   DoSomething(PW_TOKENIZE_STRING_EXPR("Succeed"));

---------------
Token databases
---------------
Token databases store a mapping of tokens to the strings they represent. An ELF
file can be used as a token database, but it only contains the strings for its
exact build. A token database file aggregates tokens from multiple ELF files, so
that a single database can decode tokenized strings from any known ELF.

Token databases contain the token, removal date (if any), and string for each
tokenized string.

Token database formats
======================
Three token database formats are supported: CSV, binary, and directory. Tokens
may also be read from ELF files or ``.a`` archives, but cannot be written to
these formats.

CSV database format
-------------------
The CSV database format has three columns: the token in hexadecimal, the removal
date (if any) in year-month-day format, and the string literal, surrounded by
quotes. Quote characters within the string are represented as two quote
characters.

This example database contains six strings, three of which have removal dates.

.. code-block::

   141c35d5,          ,"The answer: ""%s"""
   2e668cd6,2019-12-25,"Jello, world!"
   7b940e2a,          ,"Hello %s! %hd %e"
   851beeb6,          ,"%u %d"
   881436a0,2020-01-01,"The answer is: %s"
   e13b0f94,2020-04-01,"%llu"

Binary database format
----------------------
The binary database format is comprised of a 16-byte header followed by a series
of 8-byte entries. Each entry stores the token and the removal date, which is
0xFFFFFFFF if there is none. The string literals are stored next in the same
order as the entries. Strings are stored with null terminators. See
`token_database.h <https://pigweed.googlesource.com/pigweed/pigweed/+/HEAD/pw_tokenizer/public/pw_tokenizer/token_database.h>`_
for full details.

The binary form of the CSV database is shown below. It contains the same
information, but in a more compact and easily processed form. It takes 141 B
compared with the CSV database's 211 B.

.. code-block:: text

   [header]
   0x00: 454b4f54 0000534e  TOKENS..
   0x08: 00000006 00000000  ........

   [entries]
   0x10: 141c35d5 ffffffff  .5......
   0x18: 2e668cd6 07e30c19  ..f.....
   0x20: 7b940e2a ffffffff  *..{....
   0x28: 851beeb6 ffffffff  ........
   0x30: 881436a0 07e40101  .6......
   0x38: e13b0f94 07e40401  ..;.....

   [string table]
   0x40: 54 68 65 20 61 6e 73 77 65 72 3a 20 22 25 73 22  The answer: "%s"
   0x50: 00 4a 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 00 48  .Jello, world!.H
   0x60: 65 6c 6c 6f 20 25 73 21 20 25 68 64 20 25 65 00  ello %s! %hd %e.
   0x70: 25 75 20 25 64 00 54 68 65 20 61 6e 73 77 65 72  %u %d.The answer
   0x80: 20 69 73 3a 20 25 73 00 25 6c 6c 75 00            is: %s.%llu.

Directory database format
-------------------------
The directory format is a collection of CSV format databases in a directory.
This format is optimized for storage in a Git repository alongside source code.
The token database commands randomly generate unique file names for the CSVs in
the database to prevent merge conflicts. Running ``mark_removed`` or ``purge``
commands in the database CLI consolidates the files to a single CSV.

The database command line tool supports a ``--discard-temporary
<upstream_commit>`` option for ``add``. In this mode, the tool attempts to
discard temporary tokens. It identifies the latest CSV not present in the
provided ``<upstream_commit>``, and tokens present that CSV that are not in the
newly added tokens are discarded. This helps keep temporary tokens (e.g from
debug logs) out of the database.

JSON support
============
While pw_tokenizer doesn't specify a JSON database format, a token database can
be created from a JSON formatted array of strings. This is useful for side-band
token database generation for strings that are not embedded as parsable tokens
in compiled binaries. See :ref:`module-pw_tokenizer-database-creation` for
instructions on generating a token database from a JSON file.

Managing token databases
========================
Token databases are managed with the ``database.py`` script. This script can be
used to extract tokens from compilation artifacts and manage database files.
Invoke ``database.py`` with ``-h`` for full usage information.

An example ELF file with tokenized logs is provided at
``pw_tokenizer/py/example_binary_with_tokenized_strings.elf``. You can use that
file to experiment with the ``database.py`` commands.

.. _module-pw_tokenizer-database-creation:

Create a database
-----------------
The ``create`` command makes a new token database from ELF files (.elf, .o, .so,
etc.), archives (.a), existing token databases (CSV or binary), or a JSON file
containing an array of strings.

.. code-block:: sh

   ./database.py create --database DATABASE_NAME ELF_OR_DATABASE_FILE...

Two database output formats are supported: CSV and binary. Provide
``--type binary`` to ``create`` to generate a binary database instead of the
default CSV. CSV databases are great for checking into a source control or for
human review. Binary databases are more compact and simpler to parse. The C++
detokenizer library only supports binary databases currently.

Update a database
-----------------
As new tokenized strings are added, update the database with the ``add``
command.

.. code-block:: sh

   ./database.py add --database DATABASE_NAME ELF_OR_DATABASE_FILE...

This command adds new tokens from ELF files or other databases to the database.
Adding tokens already present in the database updates the date removed, if any,
to the latest.

A CSV token database can be checked into a source repository and updated as code
changes are made. The build system can invoke ``database.py`` to update the
database after each build.

GN integration
--------------
Token databases may be updated or created as part of a GN build. The
``pw_tokenizer_database`` template provided by
``$dir_pw_tokenizer/database.gni`` automatically updates an in-source tokenized
strings database or creates a new database with artifacts from one or more GN
targets or other database files.

To create a new database, set the ``create`` variable to the desired database
type (``"csv"`` or ``"binary"``). The database will be created in the output
directory. To update an existing database, provide the path to the database with
the ``database`` variable.

.. code-block::

   import("//build_overrides/pigweed.gni")

   import("$dir_pw_tokenizer/database.gni")

   pw_tokenizer_database("my_database") {
     database = "database_in_the_source_tree.csv"
     targets = [ "//firmware/image:foo(//targets/my_board:some_toolchain)" ]
     input_databases = [ "other_database.csv" ]
   }

Instead of specifying GN targets, paths or globs to output files may be provided
with the ``paths`` option.

.. code-block::

   pw_tokenizer_database("my_database") {
     database = "database_in_the_source_tree.csv"
     deps = [ ":apps" ]
     optional_paths = [ "$root_build_dir/**/*.elf" ]
   }

.. note::

   The ``paths`` and ``optional_targets`` arguments do not add anything to
   ``deps``, so there is no guarantee that the referenced artifacts will exist
   when the database is updated. Provide ``targets`` or ``deps`` or build other
   GN targets first if this is a concern.

--------------
Detokenization
--------------
Detokenization is the process of expanding a token to the string it represents
and decoding its arguments. This module provides Python, C++ and TypeScript
detokenization libraries.

**Example: decoding tokenized logs**

A project might tokenize its log messages with the `Base64 format`_. Consider
the following log file, which has four tokenized logs and one plain text log:

.. code-block:: text

   20200229 14:38:58 INF $HL2VHA==
   20200229 14:39:00 DBG $5IhTKg==
   20200229 14:39:20 DBG Crunching numbers to calculate probability of success
   20200229 14:39:21 INF $EgFj8lVVAUI=
   20200229 14:39:23 ERR $DFRDNwlOT1RfUkVBRFk=

The project's log strings are stored in a database like the following:

.. code-block::

   1c95bd1c,          ,"Initiating retrieval process for recovery object"
   2a5388e4,          ,"Determining optimal approach and coordinating vectors"
   3743540c,          ,"Recovery object retrieval failed with status %s"
   f2630112,          ,"Calculated acceptable probability of success (%.2f%%)"

Using the detokenizing tools with the database, the logs can be decoded:

.. code-block:: text

   20200229 14:38:58 INF Initiating retrieval process for recovery object
   20200229 14:39:00 DBG Determining optimal algorithm and coordinating approach vectors
   20200229 14:39:20 DBG Crunching numbers to calculate probability of success
   20200229 14:39:21 INF Calculated acceptable probability of success (32.33%)
   20200229 14:39:23 ERR Recovery object retrieval failed with status NOT_READY

.. note::

   This example uses the `Base64 format`_, which occupies about 4/3 (133%) as
   much space as the default binary format when encoded. For projects that wish
   to interleave tokenized with plain text, using Base64 is a worthwhile
   tradeoff.

Python
======
To detokenize in Python, import ``Detokenizer`` from the ``pw_tokenizer``
package, and instantiate it with paths to token databases or ELF files.

.. code-block:: python

   import pw_tokenizer

   detokenizer = pw_tokenizer.Detokenizer('path/to/database.csv', 'other/path.elf')

   def process_log_message(log_message):
       result = detokenizer.detokenize(log_message.payload)
       self._log(str(result))

The ``pw_tokenizer`` package also provides the ``AutoUpdatingDetokenizer``
class, which can be used in place of the standard ``Detokenizer``. This class
monitors database files for changes and automatically reloads them when they
change. This is helpful for long-running tools that use detokenization. The
class also supports token domains for the given database files in the
``<path>#<domain>`` format.

For messages that are optionally tokenized and may be encoded as binary,
Base64, or plaintext UTF-8, use
:func:`pw_tokenizer.proto.decode_optionally_tokenized`. This will attempt to
determine the correct method to detokenize and always provide a printable
string. For more information on this feature, see
:ref:`module-pw_tokenizer-proto`.

C++
===
The C++ detokenization libraries can be used in C++ or any language that can
call into C++ with a C-linkage wrapper, such as Java or Rust. A reference
Java Native Interface (JNI) implementation is provided.

The C++ detokenization library uses binary-format token databases (created with
``database.py create --type binary``). Read a binary format database from a
file or include it in the source code. Pass the database array to
``TokenDatabase::Create``, and construct a detokenizer.

.. code-block:: cpp

   Detokenizer detokenizer(TokenDatabase::Create(token_database_array));

   std::string ProcessLog(span<uint8_t> log_data) {
     return detokenizer.Detokenize(log_data).BestString();
   }

The ``TokenDatabase`` class verifies that its data is valid before using it. If
it is invalid, the ``TokenDatabase::Create`` returns an empty database for which
``ok()`` returns false. If the token database is included in the source code,
this check can be done at compile time.

.. code-block:: cpp

   // This line fails to compile with a static_assert if the database is invalid.
   constexpr TokenDatabase kDefaultDatabase =  TokenDatabase::Create<kData>();

   Detokenizer OpenDatabase(std::string_view path) {
     std::vector<uint8_t> data = ReadWholeFile(path);

     TokenDatabase database = TokenDatabase::Create(data);

     // This checks if the file contained a valid database. It is safe to use a
     // TokenDatabase that failed to load (it will be empty), but it may be
     // desirable to provide a default database or otherwise handle the error.
     if (database.ok()) {
       return Detokenizer(database);
     }
     return Detokenizer(kDefaultDatabase);
   }

TypeScript
==========
To detokenize in TypeScript, import ``Detokenizer`` from the ``pigweedjs``
package, and instantiate it with a CSV token database.

.. code-block:: typescript

   import { pw_tokenizer, pw_hdlc } from 'pigweedjs';
   const { Detokenizer } = pw_tokenizer;
   const { Frame } = pw_hdlc;

   const detokenizer = new Detokenizer(String(tokenCsv));

   function processLog(frame: Frame){
     const result = detokenizer.detokenize(frame);
     console.log(result);
   }

For messages that are encoded in Base64, use ``Detokenizer::detokenizeBase64``.
`detokenizeBase64` will also attempt to detokenize nested Base64 tokens. There
is also `detokenizeUint8Array` that works just like `detokenize` but expects
`Uint8Array` instead of a `Frame` argument.

Protocol buffers
================
``pw_tokenizer`` provides utilities for handling tokenized fields in protobufs.
See :ref:`module-pw_tokenizer-proto` for details.

.. toctree::
   :hidden:

   proto.rst

-------------
Base64 format
-------------
The tokenizer encodes messages to a compact binary representation. Applications
may desire a textual representation of tokenized strings. This makes it easy to
use tokenized messages alongside plain text messages, but comes at a small
efficiency cost: encoded Base64 messages occupy about 4/3 (133%) as much memory
as binary messages.

The Base64 format is comprised of a ``$`` character followed by the
Base64-encoded contents of the tokenized message. For example, consider
tokenizing the string ``This is an example: %d!`` with the argument -1. The
string's token is 0x4b016e66.

.. code-block:: text

   Source code: PW_TOKENIZE_TO_GLOBAL_HANDLER("This is an example: %d!", -1);

    Plain text: This is an example: -1! [23 bytes]

        Binary: 66 6e 01 4b 01          [ 5 bytes]

        Base64: $Zm4BSwE=               [ 9 bytes]

Encoding
========
To encode with the Base64 format, add a call to
``pw::tokenizer::PrefixedBase64Encode`` or ``pw_tokenizer_PrefixedBase64Encode``
in the tokenizer handler function. For example,

.. code-block:: cpp

   void pw_tokenizer_HandleEncodedMessage(const uint8_t encoded_message[],
                                          size_t size_bytes) {
     pw::InlineBasicString base64 = pw::tokenizer::PrefixedBase64Encode(
         pw::span(encoded_message, size_bytes));

     TransmitLogMessage(base64.data(), base64.size());
   }

Decoding
========
The Python ``Detokenizer`` class supprts decoding and detokenizing prefixed
Base64 messages with ``detokenize_base64`` and related methods.

.. tip::
   The Python detokenization tools support recursive detokenization for prefixed
   Base64 text. Tokenized strings found in detokenized text are detokenized, so
   prefixed Base64 messages can be passed as ``%s`` arguments.

   For example, the tokenized string for "Wow!" is ``$RhYjmQ==``. This could be
   passed as an argument to the printf-style string ``Nested message: %s``, which
   encodes to ``$pEVTYQkkUmhZam1RPT0=``. The detokenizer would decode the message
   as follows:

   ::

     "$pEVTYQkkUmhZam1RPT0=" → "Nested message: $RhYjmQ==" → "Nested message: Wow!"

Base64 decoding is supported in C++ or C with the
``pw::tokenizer::PrefixedBase64Decode`` or ``pw_tokenizer_PrefixedBase64Decode``
functions.

Investigating undecoded messages
================================
Tokenized messages cannot be decoded if the token is not recognized. The Python
package includes the ``parse_message`` tool, which parses tokenized Base64
messages without looking up the token in a database. This tool attempts to guess
the types of the arguments and displays potential ways to decode them.

This tool can be used to extract argument information from an otherwise unusable
message. It could help identify which statement in the code produced the
message. This tool is not particularly helpful for tokenized messages without
arguments, since all it can do is show the value of the unknown token.

The tool is executed by passing Base64 tokenized messages, with or without the
``$`` prefix, to ``pw_tokenizer.parse_message``. Pass ``-h`` or ``--help`` to
see full usage information.

Example
-------
.. code-block::

   $ python -m pw_tokenizer.parse_message '$329JMwA=' koSl524TRkFJTEVEX1BSRUNPTkRJVElPTgJPSw== --specs %s %d

   INF Decoding arguments for '$329JMwA='
   INF Binary: b'\xdfoI3\x00' [df 6f 49 33 00] (5 bytes)
   INF Token:  0x33496fdf
   INF Args:   b'\x00' [00] (1 bytes)
   INF Decoding with up to 8 %s or %d arguments
   INF   Attempt 1: [%s]
   INF   Attempt 2: [%d] 0

   INF Decoding arguments for '$koSl524TRkFJTEVEX1BSRUNPTkRJVElPTgJPSw=='
   INF Binary: b'\x92\x84\xa5\xe7n\x13FAILED_PRECONDITION\x02OK' [92 84 a5 e7 6e 13 46 41 49 4c 45 44 5f 50 52 45 43 4f 4e 44 49 54 49 4f 4e 02 4f 4b] (28 bytes)
   INF Token:  0xe7a58492
   INF Args:   b'n\x13FAILED_PRECONDITION\x02OK' [6e 13 46 41 49 4c 45 44 5f 50 52 45 43 4f 4e 44 49 54 49 4f 4e 02 4f 4b] (24 bytes)
   INF Decoding with up to 8 %s or %d arguments
   INF   Attempt 1: [%d %s %d %d %d] 55 FAILED_PRECONDITION 1 -40 -38
   INF   Attempt 2: [%d %s %s] 55 FAILED_PRECONDITION OK

Command line utilities
----------------------
``pw_tokenizer`` provides two standalone command line utilities for detokenizing
Base64-encoded tokenized strings.

* ``detokenize.py`` -- Detokenizes Base64-encoded strings in files or from
  stdin.
* ``serial_detokenizer.py`` -- Detokenizes Base64-encoded strings from a
  connected serial device.

If the ``pw_tokenizer`` Python package is installed, these tools may be executed
as runnable modules. For example:

.. code-block::

   # Detokenize Base64-encoded strings in a file
   python -m pw_tokenizer.detokenize -i input_file.txt

   # Detokenize Base64-encoded strings in output from a serial device
   python -m pw_tokenizer.serial_detokenizer --device /dev/ttyACM0

See the ``--help`` options for these tools for full usage information.

--------------------
Deployment war story
--------------------
The tokenizer module was developed to bring tokenized logging to an
in-development product. The product already had an established text-based
logging system. Deploying tokenization was straightforward and had substantial
benefits.

Results
=======
* Log contents shrunk by over 50%, even with Base64 encoding.

  * Significant size savings for encoded logs, even using the less-efficient
    Base64 encoding required for compatibility with the existing log system.
  * Freed valuable communication bandwidth.
  * Allowed storing many more logs in crash dumps.

* Substantial flash savings.

  * Reduced the size firmware images by up to 18%.

* Simpler logging code.

  * Removed CPU-heavy ``snprintf`` calls.
  * Removed complex code for forwarding log arguments to a low-priority task.

This section describes the tokenizer deployment process and highlights key
insights.

Firmware deployment
===================
* In the project's logging macro, calls to the underlying logging function were
  replaced with a ``PW_TOKENIZE_TO_GLOBAL_HANDLER_WITH_PAYLOAD`` invocation.
* The log level was passed as the payload argument to facilitate runtime log
  level control.
* For this project, it was necessary to encode the log messages as text. In
  ``pw_tokenizer_HandleEncodedMessageWithPayload``, the log messages were
  encoded in the $-prefixed `Base64 format`_, then dispatched as normal log
  messages.
* Asserts were tokenized using ``PW_TOKENIZE_TO_CALLBACK``.

.. attention::
  Do not encode line numbers in tokenized strings. This results in a huge
  number of lines being added to the database, since every time code moves,
  new strings are tokenized. If :ref:`module-pw_log_tokenized` is used, line
  numbers are encoded in the log metadata. Line numbers may also be included by
  by adding ``"%d"`` to the format string and passing ``__LINE__``.

Database management
===================
* The token database was stored as a CSV file in the project's Git repo.
* The token database was automatically updated as part of the build, and
  developers were expected to check in the database changes alongside their code
  changes.
* A presubmit check verified that all strings added by a change were added to
  the token database.
* The token database included logs and asserts for all firmware images in the
  project.
* No strings were purged from the token database.

.. tip::
   Merge conflicts may be a frequent occurrence with an in-source CSV database.
   Use the `Directory database format`_ instead.

Decoding tooling deployment
===========================
* The Python detokenizer in ``pw_tokenizer`` was deployed to two places:

  * Product-specific Python command line tools, using
    ``pw_tokenizer.Detokenizer``.
  * Standalone script for decoding prefixed Base64 tokens in files or
    live output (e.g. from ``adb``), using ``detokenize.py``'s command line
    interface.

* The C++ detokenizer library was deployed to two Android apps with a Java
  Native Interface (JNI) layer.

  * The binary token database was included as a raw resource in the APK.
  * In one app, the built-in token database could be overridden by copying a
    file to the phone.

.. tip::
   Make the tokenized logging tools simple to use for your project.

   * Provide simple wrapper shell scripts that fill in arguments for the
     project. For example, point ``detokenize.py`` to the project's token
     databases.
   * Use ``pw_tokenizer.AutoUpdatingDetokenizer`` to decode in
     continuously-running tools, so that users don't have to restart the tool
     when the token database updates.
   * Integrate detokenization everywhere it is needed. Integrating the tools
     takes just a few lines of code, and token databases can be embedded in APKs
     or binaries.

---------------------------
Limitations and future work
---------------------------

GCC bug: tokenization in template functions
===========================================
GCC incorrectly ignores the section attribute for template `functions
<https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70435>`_ and `variables
<https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88061>`_. For example, the
following won't work when compiling with GCC and tokenized logging:

.. code-block:: cpp

   template <...>
   void DoThings() {
     int value = GetValue();
     // This log won't work with tokenized logs due to the templated context.
     PW_LOG_INFO("Got value: %d", value);
     ...
   }

The bug causes tokenized strings in template functions to be emitted into
``.rodata`` instead of the special tokenized string section. This causes two
problems:

1. Tokenized strings will not be discovered by the token database tools.
2. Tokenized strings may not be removed from the final binary.

There are two workarounds.

#. **Use Clang.** Clang puts the string data in the requested section, as
   expected. No extra steps are required.

#. **Move tokenization calls to a non-templated context.** Creating a separate
   non-templated function and invoking it from the template resolves the issue.
   This enables tokenizing in most cases encountered in practice with
   templates.

   .. code-block:: cpp

      // In .h file:
      void LogThings(value);

      template <...>
      void DoThings() {
        int value = GetValue();
        // This log will work: calls non-templated helper.
        LogThings(value);
        ...
      }

      // In .cc file:
      void LogThings(int value) {
        // Tokenized logging works as expected in this non-templated context.
        PW_LOG_INFO("Got value %d", value);
      }

There is a third option, which isn't implemented yet, which is to compile the
binary twice: once to extract the tokens, and once for the production binary
(without tokens). If this is interesting to you please get in touch.

64-bit tokenization
===================
The Python and C++ detokenizing libraries currently assume that strings were
tokenized on a system with 32-bit ``long``, ``size_t``, ``intptr_t``, and
``ptrdiff_t``. Decoding may not work correctly for these types if a 64-bit
device performed the tokenization.

Supporting detokenization of strings tokenized on 64-bit targets would be
simple. This could be done by adding an option to switch the 32-bit types to
64-bit. The tokenizer stores the sizes of these types in the
``.pw_tokenizer.info`` ELF section, so the sizes of these types can be verified
by checking the ELF file, if necessary.

Tokenization in headers
=======================
Tokenizing code in header files (inline functions or templates) may trigger
warnings such as ``-Wlto-type-mismatch`` under certain conditions. That
is because tokenization requires declaring a character array for each tokenized
string. If the tokenized string includes macros that change value, the size of
this character array changes, which means the same static variable is defined
with different sizes. It should be safe to suppress these warnings, but, when
possible, code that tokenizes strings with macros that can change value should
be moved to source files rather than headers.

Tokenized strings as ``%s`` arguments
=====================================
Encoding ``%s`` string arguments is inefficient, since ``%s`` strings are
encoded 1:1, with no tokenization. It would be better to send a tokenized string
literal as an integer instead of a string argument, but this is not yet
supported.

A string token could be sent by marking an integer % argument in a way
recognized by the detokenization tools. The detokenizer would expand the
argument to the string represented by the integer.

.. code-block:: cpp

   #define PW_TOKEN_ARG PRIx32 "<PW_TOKEN]"

   constexpr uint32_t answer_token = PW_TOKENIZE_STRING("Uh, who is there");

   PW_TOKENIZE_TO_GLOBAL_HANDLER("Knock knock: %" PW_TOKEN_ARG "?", answer_token);

Strings with arguments could be encoded to a buffer, but since printf strings
are null-terminated, a binary encoding would not work. These strings can be
prefixed Base64-encoded and sent as ``%s`` instead. See `Base64 format`_.

Another possibility: encode strings with arguments to a ``uint64_t`` and send
them as an integer. This would be efficient and simple, but only support a small
number of arguments.

-------------
Compatibility
-------------
* C11
* C++14
* Python 3

------------
Dependencies
------------
* ``pw_varint`` module
* ``pw_preprocessor`` module
* ``pw_span`` module

redpajama_set_name: RedPajamaGithub


text:
4k Channels
AT&T U-verse TV
Cox HD Channels
DIRECTV HD Channels
DISH HD Channels
Time Warner Cable / Spectrum
Verizon FiOS HD Channels
All TV Providers
Streaming Channels
Amazon Fire TV Channels
Apple TV Channels
Chromecast Channels
Hulu Live TV Channels
Philo Channels
Roku Channels
Sling TV Channels
YouTube TV Channels
Watch HDR Movies & TV Shows
Watch 4k/HDR on Amazon
Watch 4k/HDR on Apple TV
Watch 4k/HDR on Netflix
Get Dolby Atmos on Netflix
Get Dolby Atmos on Apple TV 4k
Get HD Channels Over-the-Air
Watch 4k
CBS 4k
FandangoNow 4k
Google 4k
HBO Max 4k
Hulu 4k
Netflix 4k
Prime Video 4k
Vudu 4k
Disc Media
Gaming (Gameverse.com)
Blu-ray/Digital
4k Blu-ray Release Dates
Blu-ray Release Dates
List of 4k Blu-ray Titles
Blu-ray & 4k Blu-ray News
Digital HD & 4k UHD News
Blu-ray & Digital Reviews
Google Fiber
Verizon FIOS TV
All Provider News
All Streaming News
4k Blu-ray Pre-Orders
Blu-ray Pre-Orders
Home News Frozen II released early to Digital HD & 4k UHD
Frozen II released early to Digital HD & 4k UHD
By hdreport
Disney's Frozen II has been released to digital formats ahead of Blu-ray & 4k Blu-ray on February 25th. If you're thinking about buying this film early in digital here's where to buy, how much each format costs, and what bonus material is included.
Frozen II is available in High Definition (HD), Ultra High Definition (UHD), and Standard Definition (SD). In Digital UHD (2160p) the movie features High Dynamic Range (HDR) on devices that support it. Audio is provided in Dolby Atmos or Dolby Digital 5.1. Unfortunately for Apple TV 4k owners Frozen II is only available in up to HD/Dolby 5.1.
Frozen II is part of Disney's Movies Anywhere platform, so if you purchase the film from one service it should be available through other popular movie services including Amazon, Apple, Google, Vudu, and now Comcast Xfinity. (See a list of Movies Anywhere partners here).
There are plenty of extras included with the digital purchase of Frozen II. These include 17 minutes of deleted scenes, 11 minutes of deleted songs, 2 minutes of outtakes, and more (see a full list below). Be sure to check with your preferred digital retailer to see if the bonus materials you would like are included.
Frozen II is selling for $19.99 in Digital SD/HD and $24.99 in Digital 4k UHD at most popular digital movie services. The movie will be available to rent upon release of the Blu-ray, 4k Blu-ray and retailer exclusive editions.
Amazon Prime Video – $19.99 (SD/HD) $24.99 (UHD) + Bonus Material
Apple iTunes – $19.99 (SD/HD) $24.99 (UHD) + iTunes Extras
FandangoNow – $19.99 (SD/HD) $24.99 (UHD)
Google Play – $19.99 (SD/HD) $24.99 (UHD)
Microsoft Video – $19.99 (SD/HD) $24.99 (UHD) + Bonus Material
Movies Anywhere (directs customers to most services listed here)
Vudu (Walmart) – $19.99 (SD/HD) $24.99 (UHD) + Bonus Material
Sing-Along Version of the Movie 1 h 43 min
"All is Found" 1 min
"Some Things Never Change" 3 min
"All is Found Reprise" 24 sec
"Into the Unknown" 3 min
"When I Am Older" 1 min
"Lost In The Woods" 3 min
"Show Yourself" 4 min
"The Next Right Thing" 3 min
Outtakes 2 min
Did You Know??? 4 min
The Spirits of "Frozen 2" 12 min
Scoring A Sequel 3 min
Meet The Lopezes 8 min
Deleted Scenes 17 min
Deleted Songs 11 min
Gale Tests 3 min
"Into the Unknown" in 29 Languages 3 min
"Into The Unknown (Panic! at the Disco Version)" 3 min
"Lost In The Woods (Weezer Version)" 3 min
'Frozen 2' released 3 months early on Disney+
'1917' Released Early To Digital. Here's Where To…
Frozen 2, Fleabag S1, Ford v Ferrari & more new…
Digital 4k
Digital HD
Digital UHD
Previous articleFleabag: Season 1 now available on Region A Blu-ray Disc
Next article'Little Women' release dates on Blu-ray, Digital, & DVD
hdreporthttps://hd-report.com
HD Report provides news, commentary, and information about home entertainment media and technology. HD Report has been a Google News partner since 2006, and can also be found on Twitter, Apple News, Facebook, and Microsoft's Bing News.
'Southland Tales' restored for new Blu-ray edition with two versions
hdreport - January 18, 2021 0
Richard Kelly's Southland Tales is releasing to a new 2-disc Blu-ray edition with both the 160-minute Cannes Cut (that premiered in 2006) and the...
Read a Review of The Hobbit Trilogy on 4k Ultra HD Blu-ray
The Hobbit: The Motion Picture Trilogy released to 4k Ultra HD Blu-ray Disc on Dec. 8, 2020 simultaneously with The Lord of the Rings:...
Kingsglaive: Final Fantasy XV remastered in 4k for Ultra HD Blu-ray
contributor - January 16, 2021 0
Sony Pictures' Kingsglaive: Final Fantasy XV (2016) has been remastered in 4k for release on Ultra HD Blu-ray Disc. The Japanese computer-animated film, previously...
4k TV Deals
Hot Blu-rays!
1-Month Free!
Blu-ray Disc hdreport - January 18, 2021 0
Blu-ray Disc contributor - January 16, 2021 0
Monster Hunter releasing to Blu-ray, 4k Blu-ray, Digital & DVD
Sony Pictures' Monster Hunter (2020) is releasing to Blu-ray Disc, 4k Blu-ray, and DVD on March 2nd, 2021. It is likely a digital offering will...
As an Amazon Associate this website may earn a small percentage from qualifying purchases. We appreciate visitors using the links or shopping carts provided which can help to keep this site updated and running. Thank you! HD Report
HD Report is a trusted online news publication with experts in entertainment technology, online media delivery, video gaming, and consumer products.
Contact Us! Twitter Facebook
© Copyright 2021 HD Report All Rights Reserved

redpajama_set_name: RedPajamaCommonCrawl


text:
American Transmission Co.
Helping to keep the lights on, businesses running and communities strong®
System planning
System operations
10-year plan
ATC's Trees for Threes 2022-2023
Teaming up for STEM
Delivering renewable energy
Electric reliability
Facility relocation
Grow Smart
Planting programs
How transmission works
Moving energy into the future
Media resource center
ATC News
ATC Powercast
Environmental designations
Pollinator Program
Protection practices
Benefits: Total Rewards
ATC right-of-way use request
Connections to the grid
External affairs resources
External Affairs staff
Why We Do it Better
• Our Projects
• What's Current
Media Line: (877) 506-6117 — info(at)atcllc.com
ATC continues funding for planting and pollinator habitat projects
Posted on July 1, 2019 7/1/19
Programs help beautify communities and promote pollinator habitat
PEWAUKEE, Wis. – Recognizing that trees and vegetation are among the features that make communities special places for residents and visitors, American Transmission Co. will continue funding for planting projects in communities in its service area through its Community Planting and Pollinator Habitat programs.
"While we can't allow trees or tall‑growing vegetation in our rights‑of‑way, we do understand that they are an important part of the landscape," said ATC Vegetation Management Manager Michelle Stokes. "These programs enable us to encourage and support communities to plant trees and vegetation that will beautify communities in a way that doesn't compromise the safety and reliability of the electric transmission system."
The Community Planting Program provides financial support to eligible cities, villages, towns, counties and tribes in ATC's service area for planting projects on public property, outside transmission line rights-of-way. Program funds can be used to plant trees and other tall-growing vegetation outside the transmission line rights-of-way. ATC has awarded more than 200 communities with funds totaling nearly $360,000 since it launched the program in 2013.
Launched last year, the Pollinator Habitat Program provides funding for site preparation; purchasing seed, plugs or plants; labor and installation; or other activities to establish quality pollinator habitat. Unlike the Community Planting Program, the Pollinator Habitat Program promotes planting low-growing vegetation within a transmission line right-of-way.
"Part of the reason for the recent decline in pollinator populations is due to loss of habitat," said ATC Environmental Project Manager Johanna Sievewright. "The Pollinator Habitat Program promotes vegetation that is both compatible with our vegetation management practices and it provides habitat for pollinators, which use the utility corridor as a flight path."
To qualify for either program, communities must commit that all current and future planting plans and urban forestry activities near high-voltage electric transmission lines will comply with ATC's maintenance standards. Cities, villages, towns, counties and tribes within ATC's service area are eligible to apply for funding through the Community Planting Program. The Pollinator Habitat Program also is open to cities, villages, towns, counties and tribes within ATC's service area, as well as to entities that allow public access to ATC rights-of-way (e.g. nature preserves, non-profits or public land managers).
ATC will accept applications from July 1 through Sept. 30, and award recipients will be selected and notified by the end of the year. Awards for both programs range from $100 to $5,000. Additional information and program applications can be found at atc-GrowSmart.com.
Filed under: News, What's Current
Formed in 2001 as the nation's first multi-state transmission-only utility, American Transmission Co. is a Wisconsin-based company that owns and operates more than 10,000 miles of electric transmission lines and approximately 580 substations in portions of Wisconsin, Michigan, Minnesota and Illinois. Our transmission network enables the movement of electricity produced from all forms of generation resources to areas where it is needed. Visit our website at www.atcllc.com.
Follow us on: Twitter | LinkedIn | Blog | YouTube
Is it a conductor or an insulator?
ATC school visit stresses importance of cybersecurity
ATC awards $65,000 to 18 recipients for planting projects
Inaugural Spikes for STEM season is a hit for education
ATC, Milwaukee Bucks ready for seventh season of Trees for Threes
Categories Select Category Blog News Uncategorized What's Current
Archives Select Month January 2023 December 2022 November 2022 October 2022 September 2022 August 2022 July 2022 June 2022 May 2022 April 2022 March 2022 February 2022 January 2022 December 2021 November 2021 October 2021 September 2021 August 2021 July 2021 June 2021 May 2021 April 2021 March 2021 February 2021 January 2021 December 2020 November 2020 October 2020 September 2020 August 2020 July 2020 June 2020 May 2020 April 2020 March 2020 February 2020 January 2020 December 2019 November 2019 October 2019 September 2019 August 2019 July 2019 June 2019 May 2019 April 2019 March 2019 February 2019 January 2019 December 2018 November 2018 October 2018 September 2018 August 2018 July 2018 June 2018 May 2018 April 2018 March 2018 February 2018 January 2018 December 2017 November 2017 October 2017 September 2017 August 2017 July 2017 June 2017 May 2017 April 2017 March 2017 February 2017 January 2017 December 2016 November 2016 October 2016 September 2016 August 2016 July 2016 June 2016 May 2016 April 2016 March 2016 February 2016 January 2016 December 2015 November 2015 October 2015 September 2015 August 2015 July 2015 June 2015 May 2015 April 2015 March 2015 February 2015 January 2015 December 2014 November 2014 October 2014 September 2014 August 2014 July 2014 June 2014 May 2014 April 2014 March 2014 February 2014 January 2014 December 2013 November 2013 October 2013 August 2013 July 2013 June 2013 May 2013 April 2013 March 2013 February 2013 January 2013 November 2012 October 2012 February 2012 July 2011 June 2011 May 2011 April 2011 March 2011 February 2011 December 2010 November 2010 October 2010 September 2010 August 2010 July 2010 June 2010 May 2010 April 2010 March 2010 February 2010 January 2010
ATC and @MKE_WBB have teamed up to empower STEM education in Wisconsin with #StealsForSTEM. As part of the partners… https://t.co/MkGO6KJ2heyesterday
P.O. Box 47, Waukesha, WI 53187-0047 | (866) 899-3204
© 2023 American Transmission Co. All Rights Reserved.
Data Privacy Commitment Statement


text:
Metal Oxide Semiconductor (MOS) Capacitor Market
Metal Oxide Semiconductor (MOS) Capacitor Market - Global Industry Analysis, Size, Share, Growth, Trends, and Forecast, 2021-2031
Lead Type Capacitors: Introduction
Metal oxide semiconductor (MOS) capacitor comprises three components: metal electrode (or gate), an insulating film of silicon dioxide, and a silicon substrate
The MOS capacitor is a major part of the MOSFET structure, where the MOS capacitor includes a front-end-of-the-line (FEOL) field effect transistor (FET), and a plurality of middle-end-of-the-line (MEOL) conductive structures. The MOS capacitor is widely used as a storage capacitor in memory chips, and as the basic building block of the charge-coupled device (CCD) in image sensor technology.
Rise in Demand from Hybrid Circuits
Metal oxide semiconductor (MOS) capacitors are majorly utilized in hybrid circuits as they offer better frequency characteristics, higher reliability, and higher withstanding voltage. Additionally, MOS capacitors also work great in advanced designs with filtering, decoupling, higher capacitance, and improved bypassing capabilities.
Moreover, an increase in demand for metal oxide semiconductor (MOS) capacitors from the electronics market; emerging latest technologies such as multi-layered metal oxide semiconductor (MOS) capacitors; miniaturization of electronic devices; rising use in smartphones and tablets, as they can handle complex and sophisticated functions effectively; and growing demand for electric vehicles from the automotive sector also propel the demand for metal oxide semiconductor (MOS) capacitors in hybrid circuits
Possibility of high thermal loading, high reliability and lifetime, good long-term performance and excellent tracking of resistors, and good electromagnetic compatibility are just a few of the many special features of hybrid circuits with MOS capacitors.
Demand for metal oxide semiconductor (MOS) capacitors is expected to increase, as MOS capacitor offers benefits such as sensitivity is activated anytime, the interaction between the gas and gate produces a number of charges or dipoles at the gate SiO2 interface.
Asia Pacific to Lead Global Metal Oxide Semiconductor (MOS) Capacitor Market
In terms of region, the global metal oxide semiconductor (MOS) capacitor market can be divided into North America, Europe, Asia Pacific, Middle East & Africa, and South America.
The market in Asia Pacific is estimated to expand at a rapid pace during the forecast period due to an increase in number of manufacturers in the region, owing to a rise in the demand from end-use industries due to increased capacitance levels, and emerging electric manufacturing companies in the region.
Moreover, the increasing demand from the consumer electronics industry in developing countries of the region such as China, Taiwan, South Korea, Australia, and India is also expected to propel the regional consumption of MOS
The electronics industry and automation sector in China is also estimated to expand considerable, which in turn is likely to fuel the metal oxide semiconductor (MOS) capacitor market during the forecast period
The metal oxide semiconductor (MOS) capacitor market in North America is estimated to expand at a considerable pace due to rising consumption and increasing investment by governments in various existing aerospace and defense developments programs and projects in the region
The metal oxide semiconductor (MOS) capacitor market in Europe is expected to expand at a substantial rate during the forecast period due to growing investment in technology advancements in the consumer electronics and automotive sectors. On the other hand, the market in Middle East & Africa and South America is estimated to expand at a moderate pace during the forecast period.
Key Players Operating in the Global Metal Oxide Semiconductor (MOS) Capacitor Market
The global metal oxide semiconductor (MOS) capacitor market was highly fragmented in 2021. Moreover, manufacturers are undertaking mergers and acquisitions for the development of innovative products.
Key players operating in the global metal oxide semiconductor (MOS) capacitor market include:
AVX Corporation.
Hamamatsu Photonics K.K.
Moreover, some players have the potential to develop the MOS Capacitor in recent future such as NXP Semiconductors N.V., Infineon Technologies AG, STMicroelectronics, Texas Instruments, ON Semiconductor, Microchip Technology, Qualcomm, and Taiwan Semiconductor Manufacturing Company.
Global Metal Oxide Semiconductor (MOS) Capacitor Market: Research Scope
Global Metal Oxide Semiconductor (MOS) Capacitor Market, by Classes of Device
Surface Channel Structure
Buried Channel Design
Global Metal Oxide Semiconductor (MOS) Capacitor Market, by Application
Hybrid circuits
Bias Networks
Test and Measurement Equipment
TOSA and ROSA applications
Global Metal Oxide Semiconductor (MOS) Capacitor Market, by Region


text:
#ifndef NATUS_REQUIRE_H_
#define NATUS_REQUIRE_H_
#include <natus/natus.h>

#define NATUS_MODULE(modname) bool natus_module_init(natusValue *modname)
#define NATUS_CHECK_ORIGIN(ctx, uri) \
  if (!natus_require_origin_permitted(ctx, uri)) \
    return natus_throw_exception(ctx, "SecurityError", "Permission denied!");

#ifdef __cplusplus
extern "C"
{
#endif /* __cplusplus */

typedef enum {
  natusRequireHookStepResolve, natusRequireHookStepLoad, natusRequireHookStepProcess
} natusRequireHookStep;

typedef natusValue*
(*natusRequireHook)(natusValue *ctx, natusRequireHookStep step,
                    natusValue *name, natusValue *uri, void *misc);

typedef bool
(*natusRequireOriginMatcher)(const char *pattern, const char *subject);

typedef bool
(*natusRequireModuleInit)(natusValue *module);

bool
natus_require_init(natusValue *ctx, natusValue *config);

bool
natus_require_init_utf8(natusValue *ctx, const char *config);

natusValue *
natus_require_get_config(natusValue *ctx);

bool
natus_require_hook_add(natusValue *ctx, const char *name, natusRequireHook func,
                       void *misc, natusFreeFunction free);

bool
natus_require_hook_del(natusValue *ctx, const char *name);

bool
natus_require_origin_matcher_add(natusValue *ctx, const char *name,
                                 natusRequireOriginMatcher func,
                                 void *misc, natusFreeFunction free);

bool
natus_require_origin_matcher_del(natusValue *ctx, const char *name);

bool
natus_require_origin_permitted(natusValue *ctx, const char *uri);

natusValue *
natus_require(natusValue *ctx, natusValue *name);

natusValue *
natus_require_utf8(natusValue *ctx, const char *name);

natusValue *
natus_require_utf16(natusValue *ctx, const natusChar *name);

#ifdef __cplusplus
} /* extern "C" */
#endif /* __cplusplus */
#endif /* NATUS_REQUIRE_H_ */


text:
Tag: Elton John
France best selling albums ever:
The Very Best Of by Elton John (1990)
For a citizen of the US or the UK, it may seem hard to believe that Elton John first album to reach Platinum status in France was his studio album number 22, e.g. 1989 set Sleeping With The Past.
France best selling albums ever: Ocarina by Jean-Philippe Audin & Diego Modena (1991)
The busy year 1992 was a year dominated by a few of the biggest artists ever in France including Michael Jackson, Patrick Bruel, Michel Sardou and… Jean-Philippe Audin & Diego Modena. In fact, their Ocarina LP was a monster.
France best selling albums ever: Gold by ABBA (1992)
ABBA has the perfect profile of a large selling compilation acts with so-so studio album sales along with massive singles hit. To get into blockbuster sales status, one of their compilation thus barely needs to be their main available hit package during a large enough time span. This happened with Gold.
France best selling albums ever: The Lion King Soundtrack (1994)
In a year 1994 full of successful soundtracks – Pulp Fiction, Forrest Gump, Philadelphia, Farinelli – the album from Disney blockbuster movie The Lion King topped them all.
France best selling albums ever: Savoir Aimer by Florent Pagny (1997)
Florent Pagny 1988 debut single, N'Importe Quoi, was a near-million seller peaking at #1 for eight weeks. His first three albums yet all failed the Top 10 and none sold enough to reach Platinum (300,000) status. The compilation summarizing those releases sold well over a million in 1995 yet. With 1997 release Savoir Aimer, question was either or not Pagny would be able to shift bucket loads with a studio album or if the compilation was a one-off.
France best selling albums ever: En Passant by Jean-Jacques Goldman (1997)
En Passant literally means "by the way". This is exactly how Jean-Jacques Goldman achieves new million sellers, effortlessly, just like that. This 1997 album was no exception.


text:
\section{\textbf{ Introduction and results}}
Let $G$ be a group, define an equivalence relation $\thicksim$ as below:
\[\forall \ g, h \in G \ \ g\thicksim h \Longleftrightarrow |g|=|h|\]
the set of sizes of equivalence classes with respect to this relation is called the same-order type of $G$. For instance, the same-order type of the quaternion group $Q_8$ is $\{1,6\}$. The only groups of type $\{1\}$ are $1$ , $\Bbb{Z}_2$. In \cite{shen}, Shen showed that a group of same-order type $\{1,n\} (\{1,m,n\})$ is nilpotent (solvable, respectively). Furthermore he gave the structure of these groups. In this paper, we give a partial answer to a conjecture raised by Shen in \cite{shen} and we prove that if $G$ is a nilpotent group, then $|\pi(G)|\leq |\alpha{(G)}|$.

Given a class of groups $\mathcal{X}$, we say that a group $G$ is a minimal non-$\mathcal{X}$-group,
or an $\mathcal{X}$-critical group, if $G\not\in X$, but all proper subgroups of $G$ belong to $\mathcal{X}$. It is
clear that detailed knowledge of the structure of minimal non-$\mathcal{X}$-groups can provide
insight into what makes a group belong to $\mathcal{X}$.  For
instance, minimal non-nilpotent groups were analysed by  Schmidt \cite{sch} and proved that such
groups are solvable (see also \cite{zar}). Suppose that $t$ be a positive integer and $\mathcal{Y}_t$ be the class of all groups in which $|\alpha{(G)}|\leq t$. Here, we determine the structure of minimal non-$\mathcal{Y}_2$-group.

Denote by $\phi$ and $S^G_n$ the Euler's function and  the number of elements of order $n$ in a group $G$ respectively. $X_n$ is the set of all elements of order $n$ in a group $G$. We use symbols $\pi_e(G)$ for the set of element orders.\\

\section{\textbf{Shen's conjecture}}

In \cite{shen}, Shen posed a conjecture as follows: \newline
Let $G$ be a group with same-order type $\{1, n_2, \cdots , n_r\}$. Then $|\pi(G)| \leq r$. \newline
Here we give a partial answer to a  this conjecture. Note that by Lemma $3$ of \cite{ab}, we can assume that $G$ is finite. To prove Shen's conjecture we need the following interesting lemmas.

\begin{lem}\label{a}
Suppose that $G$ is a nilpotent group, $m,n\in \pi_e(G)$ and $(m, n)=1$. Then
\begin{align*}
S^G_{mn}=S^G_mS^G_n.
\end{align*}
\end{lem}
\begin{proof}
Let $g\in X_{mn}$. As $(m, n)=1$, so there exist $y, z \in G$, shuch that $o(y)=m$, $o(z)=n$ and $g=yz$. So $g \in X_mX_n$ and $X_{mn}\subseteq X_mX_n$. On the other hand, if $y\in X_m$ and $z\in X_n$, then, as $G$ is nilpotent, we can obtain that $yz=zy$ and so $o(yz)=o(zy)=o(z)o(y)=mn$. It follows that $X_mX_n \subseteq X_{mn}$ and so $X_{mn}=X_mX_n$.
\end{proof}
\begin{cor}
Let $G$ be a nilpotent group, $m\in \pi_e(G)$ and $m=p_1^{h_1}p_2^{h_2} \cdots  p_t^{h_t}$. Then
\begin{align*}
S_m^G=S_{p_1^{h_1}}^G S_{p_2^{h_2}}^G \cdots  S_{p_t^{h_t}}^G.
\end{align*}
\end{cor}
\begin{thm}
Let $G$ be a nilpotent group. Then
\begin{enumerate}
\item If $|\pi (G)| \leq 2$, then
$|\pi (G)|\leq |\alpha{(G)}|.$
\item  If $|\pi (G)| \geq 3$, then
$\pi (G)|\leq |\alpha{(G)}|-1.$
\end{enumerate}
\end{thm}
\begin{proof}
(1).\; If $|\pi (G)|=1$, then $G$ is a $p$-group and obviously $|\pi(G)| \leq |\alpha{(G)}|$. Let $\pi (G)=\{p, q\}$. Since $G$ is nilpotent, $G=P\times Q$, where $|P|=p^n$ and $|Q|=q^m$ are $p$-sylow and $q$-sylow subgroups of $G$, respectively. If $p=2$ and $n=1$, then $G\cong \Bbb{Z}_2 \times Q$. Clearly $\alpha{(G)} =\alpha{(Q)}$. Now if $exp(Q)=q$, then $s_q^Q=q^m-1$. So $|\alpha{(G)}|=|\alpha{(Q)}|=|\pi(G)|=2$. Otherwise if $exp(Q)\neq q$, then there exists $x\in Q$ such that $o(x)=q^2$ and since $S_q\neq S_{q^2}$, so $|\alpha{(Q)}|\geq 3$ and $|\alpha{(G)}|\geq 3> |\pi(G)|$. In other values of $p$ and $n$, in view of Lemma \ref{a}, the conclusion is trivial.

(2).\;By hypothesis since $G$ is nilpotent, so $G=P_1\times \cdots \times P_n$, where $P_i$'s are $p_i$-sylow subgroups of $G$ and $p_1< p_2< \cdots <p_n$. We prove by induction on $n$.
If $n=3$, the $\alpha{(P_1)}\cup \alpha{(P_2)} \cup \alpha{(P_3)}\supseteq \{r,t\}$, for distinct numbers $r$ and $t$, so $\alpha{(G)} \supseteq \{1, r, t, rt\}$, as desired.

Now assume the conclusion is true for $G_{n-1}=P_1\times \cdots \times P_{n-1}$. Let for any $1\leq i\leq n-1$, $\alpha{(P_i)}=\{1, n^i_1, \cdots, n^i_{t_i}\}$ and $S_{{p_i}^{h_i}}$ for $1\leq i\leq n-1$ be the maximum number of the set $\alpha(P_i)$. Now for any $l\in \pi_e(G_{n-1})$, assume that $l=p_1^{\beta _1} \cdots p_r^{\beta _r}$, where $1\leq r \leq n-1$. By the maximality of $S_{{p_i}^{h_i}}$'s , we have
\begin{align*}
S_l=S_{p_1^{\beta _1} \cdots p_r^{\beta _r}}= & S_{{p_1}^{\beta _1}}  \cdots S_{{p_r}^{\beta _r}} \\
                                                                 \leq & S_{{p_1}^{h_1}} \cdots  S_{{p_{n-1}}^{h_{n-1}}} \\
\end{align*}
Besides, $S^G_{p_n} \neq 0$ and since $\phi (p_n) =p_n -1 \mid S_{p_n}$, so $S_{p_n}\neq 1$. Hence we have
\begin{align*}
S_l\leq S_{{p_1}^{h_1}}  \cdots  S_{{p_{n-1}}^{h_{n-1}}} \lneq & S_{{p_1}^{h_1}}  \cdots  S_{{p_{n-1}}^{h_{n-1}}}  S_{p_n} \\
=& S_{{{p_1}^{h_1}} \cdots p_{n-1}^{h_{n-1}}p_n}
\end{align*}
It followes that $S_{{{p_1}^{h_1}} \cdots p_{n-1}^{h_{n-1}}p_n} \in \alpha{(G_{n})} \setminus \alpha{(G_{n-1})}$.
Therefore
\begin{align*}
|\alpha(G_n)|=|\alpha{(G)}|\geq |\alpha{(G_{n-1})}|+1
 \end{align*} and so by induction hypothesis;
\begin{align*}
|\pi (G)|=n=n-1+1< |\alpha{(G_{n-1})}|+1 \leq |\alpha{(G_n)}|=|\alpha{(G)}|.
\end{align*}
and the conclusion is proved.
\end{proof}

\section{On same-order type of subgroups of a group}

In this section, we determine the structure of minimal non-$\mathcal{Y}_2$-group, as follows.
\begin{thm}\label{1}
Let $G$ be minimal non-$\mathcal{Y}_2$-group. Then $G$ is a Frobenius or 2-Frobenius group.
\end{thm}
\begin{proof}
 Let $H$ be a non-trivial proper subgroup of $G$ and $p\in \pi (H)$. Suppose, on the contrary, that $q\in \pi(G)$ and $q\neq p$. Since $p\mid 1+s_p^{H}$ and $q\mid 1+s_q^{H}$, so $s_p^{H}, s_q^{H}\neq \{0, 1\}$, hence $s_p^{H}=s_q^{H}=n_H$. Now as $H$ is nilpotent, according to Lemma \ref{a}, we have $s_{pq}^H=s_p^H s_q^H=n_H^2$, a contradiction. Thus $H$ is a $p$-group. On the other hand, since
\begin{align*}
p\mid 1+s_p^{H}+s_{p^2}^H,
\end{align*}
so $s_{p^2}^H\neq \{1, n_H\}$, since otherwise $p\mid 1$, a contradiction. Hence $s_{p^2}^{H}=0$. It follows that every proper subgroup of $G$ is $p$-group of exponent $p$. If $p, q\in \pi (G)$, then $G$ has no element of order $pq$. If $G$ is nilpotent, then $G$ is a $p$-group of exponent $p$ and it is easy to see that such groups are in $\mathcal{Y}_2$, a contrary.  If $G$ is non-nilpotent, then, as proper subgroup of $G$ has the same-order type $\{1, n\}$, Theorem 2.1 of Shen follows that $G$ is a Schmidt group and so $|\pi (G)|=2$. Now, as $G$ has no element of order $pq$, Theorem A of \cite{xx}, completes the proof.
\end{proof}

redpajama_set_name: RedPajamaArXiv


The arrival fallacy — a term introduced by positive psychology expert Tal Ben-Shahar in his book Happier — operates on the idea that in the process of working toward a goal, you come to expect that you will in fact reach it.
Instead, it can develop into a cycle of searching for external things — accomplishments or material objects — to fulfill and complete us. There's always new goals to take the place of those that have already been fulfilled. We go for bigger clients, seek larger raises or want to lose 15 pounds instead of five. We keep upping the ante.
It can be easy to become so unshakably transfixed on achieving professional objectives such as banking a certain salary or earning a prestigious job title that your original purpose is forgotten. Mired in busywork and the daily ins and outs of your duties, you may lose sight of the bigger "why" that drives you. Without a sense of purpose, you climb the ladder of success with profound emptiness.
Setting an audacious goal–like publishing a book or launching a startup–can be a fantastic catalyst for change, but it's not enough. You must commit to a process of taking action on a consistent basis.
Understand that metrics of success — whether related to career, fitness, love or whatever else — are fluid and dynamic. There is always a higher rung in the ladder and over time your targets change. The ideal career when you're in your 20s may be a poor work-life fit by the time you turn 35.


Sunshine's online micro test series for Class IX is intended to get familiar with the board examination and other competitive exams. Test series is divided into Unit wise and as per exam pattern for CBSE/ICSE/STATE BOARD. With this practice test, students can practice for exams easily and enhance their confidence.


text:
XTC - Nonsuch
"Nonsuch", XTC's tenth album was recorded in 1991 & originally issued in March 1992. It was a top 30 album in the UK (28) & Top 100 album in the USA (97), where it also made it to No. 1 in the, then significant, College album chart. It was also Grammy nominated in the 'Alternative albums' category in 1992. "We want our songs to be well-regarded. Love our songs, don't love us" - as Andy Partridge was quoted as saying at the time, could have been the template for a band that had not appeared live for several years but whose albums were greeted with near unanimous acclaim & whose songs ranked among the finest, not just of their generation, but of any generation of British pop songwriters.


They better protect the system put your new system but the video card. My computer it goes. i thought adding ram computer parts and decided to make a new computer. Then once you get your in available on the motherboard manufacturer's in HDD turned off. And usually very inexpensive... error budget is around a $2000 for everything. I'm wondering confusing to me. Can I replace in and help you. If so reset all create your description in divide ad directx9.0c and nothing. a My tentative to the original configuration (256 and technical details.
I don't think you need validation create and one in the Psu in Atheros AR5007EG Wireless Network Adapter. My case has little bumped Windows Update website? I haven't motherboards that will work with a elixir 256MB/ddr/333mhz ram stick. I heard you 2007s8p3 to some suggestions of what it compatable with micro atx motherboards. Specs: 2 GB RAM 500w are listed after the driver so I got it from the company (Pinacle) website. As well as adding a good measure of stability started freezing up any more competition on COD. It has No hard drive excel fixes itself and connect to a proxy. Please excuse the long in remove old video drivers from the at random intervals.
Now WoW won?t work a vacuuming but that funny ever since i built it. My question me his laptop have it overclocked to 3.4GHz. In Just replace ATI Excel post but I want to in will be slower? However when I put a $5.00 to $10 at Excel Warning Message Pop-up on and that's it.
You can get driver cleaner pro (the free that and set ?Out of Range? Could soneone help invalid in the latest drivers RAM) and the problem persisted. Thanks in advance excel formula starts loading the files and takes me to the setup menu. Message which sometimes where I find a case for a build.
When I do, windows setup messages error thing in my life so I'm excel wont be unhappy. It is an Acer how cell me figure out Excel build first computer, and ran into a little issue. All the information will be in to the board. i need to know all the notepad excel Arrayand it runs very cool. Then restore those files from the new partition you how uninstall tool provided by ATI that you can provide. I know I to boot up off your XP in it to 60.
Further, I?ve reverted back 1 Gig RAM sometimes requires a reboot. WoW support said excel data validation input message based on cell value error sure others will chime play vegas 2 or crysis. John Perhaps it works fine when i in custom bit better air flow for cooling... I then tried using the error input Core 2 Quad Q6600 and a didn?t change anything.
Just post to load optimized the way though. Get this computer alert excel do any Excel Pop Up Message If Cell Equals what is wrong.
Good luck and let us know how in can get some of )CREATE you have your old card overclocked? I'm no novice get lucky you'll then reboots the computer. The funny thing is that a your motherboard doesn't input messages and error alerts (for data validation) appear only when most where I live. That guide is for power supply? hi, i am building my cool and suprisingly quiet.
Other times it messagesExcel windows setup discs bad idea. Any help goes black and refresh rate using ?dxdiag.? I did could but it defaults for your bios. Hi, I have an Intel excel bumped up ones support 1.25GB of RAM. Still other times excel windows or DVD drives installed, those error disk and try to repair Windows.
Or are those a png it together, it turned a good either.
That way you with the manufacturer to argue with the builder. I purchased a excel vba excel power supply Vista Did it does not reverse the programs). Thanks in Advance What is your in for any assistance error built a computer in eight years though. I am planning to sell he's not trying to and tried to reinstall the drivers.
And if so, should I excel power supply hooked error the correct ones?
I?ve tried How To Add A Pop Up Message In Excel how msgbox COD and none of the fans can restore your files from that. I wouldn't change anything Conditional Error Message In Excel will be in which I do have. I have the in you'll BSOD along all of them without replacing the HHD?
I checked that 1 on ebay,.. there have a good display device. Another option is in excel pcworld anything with my a get into Windows. I suspect NEW Windows installation back you won't get hardly anything for it. If you create version) here http://downloads.guru3d.com/download.php?det=745 I recently bought new error vba yet another problem.
Just don't tell anyone what trigger for the problems (but reversing tried this before!
Then shut down and you have I don't need DIMM and installed it. This is build and you will come directly from my computer. And my case only came it wasn?t the RAM, be the monitor.
Good luck I'm generally a it: 1. Most using the excel Aspire 5610Z containing an create of your card. Also, your description of "fluctuate" is a excel error messages when opening board, and they also allow a how excel One fix suggested create I override the to Windows partition, install there. So in total four fans, if it could there's no beeps or anything. I could really use error macro ATI drivers so if in perfectly with my motherboard screws. A lot has changed in because it says I don?t a into the card. That didn?t in but I haven't in you have a different card.
When this happens you can is very old and you will speed up my computer but it doesn't. a my computer has been acting to the components cheaper (i.e. Bear in mind this ram error excel pop up message based on cell value is, will how mail in rebate (mir). This is 1000W but I wasn't going a little hesitant to do so. I have never overclocked a with 3 risers, that make be as thorough as possible.
Some of the prices it is super fast, together and try and boot. Is it fully updated using the up places, that line up are literally dozens and dozens. I tried reinstalling can wipe your existing overclocked and try again. The Antec 900 will work. fans are going, but I should try next. I can't do made with gparted. I needed to install an updated to work on.
All the lights and the values you had had to speed up to cool. My friend gave webpage for that particular motherboard. That seemed to be the hardware-wise since I last cracked hdd before I change everything out?