text:
Q: What does 'linear predictor' mean within the context of a non-linear proportional hazards analysis using Generalized Additive Models (GAM)? I am conducting non-linear survival analysis using Wood's mgcv package in R. The model produces a linear predictor as the outcome variable when I produce a chart (as per the below code). What does linear predcitor actually mean? How do I interpret it?
df <- structure(list(SurvYear =c(3L, 2L, 3L, 6L, 8L, 3L, 5L, 2L, 9L,
8L, 1L, 7L, 1L, 4L, 6L, 8L, 2L, 5L, 1L, 1L, 7L, 1L, 5L, 3L, 2L,
1L, 9L, 1L, 5L, 2L, 2L, 1L, 2L, 3L, 4L, 8L, 7L, 2L, 2L, 6L, 9L,
7L, 3L, 9L, 6L, 8L, 2L, 8L, 2L, 1L, 1L, 6L, 5L, 3L, 3L, 7L, 2L,
4L, 5L, 2L, 3L, 7L, 4L, 1L, 2L, 2L, 3L, 5L, 1L, 9L, 2L, 2L, 3L,
9L, 6L, 2L, 2L, 4L, 3L, 1L, 9L, 7L, 3L, 1L, 2L, 1L, 6L, 3L, 1L,
5L, 6L, 5L, 6L, 4L, 2L, 1L, 3L, 1L, 1L, 3L, 4L, 3L, 8L, 9L, 7L,
6L, 3L, 5L, 2L, 7L, 9L, 2L, 1L, 1L, 1L, 1L, 2L, 1L, 2L, 9L, 1L,
4L, 8L, 1L, 8L, 1L, 1L, 8L, 5L, 2L, 9L, 4L, 8L, 4L, 9L, 2L, 2L,
3L, 2L, 9L, 3L, 2L, 1L, 3L, 2L, 1L, 9L, 9L, 2L, 1L, 1L, 1L, 2L,
9L, 1L, 5L, 1L, 6L, 9L, 3L, 2L, 2L, 5L, 7L, 4L, 2L, 7L, 2L, 4L,
5L, 3L, 3L, 9L, 2L, 6L, 1L, 3L, 4L, 5L, 9L, 8L, 1L, 2L, 8L, 2L,
9L, 1L, 7L, 3L, 3L, 1L, 6L, 3L, 4L, 9L, 1L, 3L, 4L, 4L, 2L, 7L,
2L, 3L, 1L, 1L, 7L, 2L, 1L, 1L, 2L, 1L, 9L, 1L, 2L, 9L, 1L, 1L,
2L, 3L, 7L, 3L, 1L, 1L, 2L, 5L, 4L, 6L, 7L, 1L, 9L, 2L, 1L, 8L,
1L, 2L, 1L, 4L, 2L, 3L, 3L, 9L, 9L, 9L, 4L, 1L, 1L, 4L, 9L, 3L,
1L, 1L, 3L, 3L, 4L, 1L, 1L, 1L, 1L, 6L, 9L, 1L, 1L, 8L, 1L, 3L,
3L, 8L, 3L, 5L, 1L, 2L, 1L, 2L, 4L, 3L, 1L, 6L, 1L, 4L, 8L, 1L,
3L, 2L, 2L, 3L, 6L, 2L, 1L, 1L, 1L, 9L, 3L, 1L, 7L, 3L, 9L, 1L,
9L, 5L, 4L), Gender = c(1L, 0L, 1L, 1L, 0L, 0L, 0L, 1L, 0L, 0L,
1L, 0L, 0L, 0L, 1L, 1L, 1L, 0L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 0L,
1L, 0L, 1L, 1L, 1L, 0L, 0L, 1L, 1L, 1L, 0L, 0L, 1L, 1L, 1L, 1L,
0L, 0L, 1L, 0L, 1L, 0L, 1L, 1L, 1L, 0L, 1L, 0L, 0L, 1L, 1L, 0L,
1L, 0L, 1L, 1L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L,
1L, 1L, 1L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L,
0L, 0L, 1L, 1L, 1L, 0L, 1L, 1L, 0L, 0L, 1L, 0L, 1L, 1L, 0L, 0L,
0L, 0L, 1L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 0L, 1L, 1L, 1L, 0L, 1L,
0L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 1L,
1L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 1L, 1L, 1L, 0L,
0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 1L, 1L, 0L,
1L, 0L, 1L, 0L, 1L, 1L, 1L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 1L, 1L,
1L, 1L, 1L, 0L, 0L, 0L, 0L, 1L, 1L, 1L, 0L, 1L, 0L, 0L, 1L, 0L,
1L, 1L, 1L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 1L,
1L, 0L, 1L, 1L, 0L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 1L, 1L, 1L,
1L, 1L, 1L, 0L, 1L, 1L, 0L, 1L, 0L, 1L, 0L, 1L, 0L, 1L, 1L, 1L,
1L, 1L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 1L, 1L, 1L, 1L, 0L, 1L, 0L,
0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 1L, 1L, 0L, 1L, 1L,
0L, 0L, 1L, 1L, 0L, 1L, 1L, 1L, 0L, 0L, 0L, 1L, 1L, 0L, 1L, 0L,
1L, 1L), Age = c(63L, 66L, 34L, 43L, 63L, 21L, 24L, 44L, 52L,
59L, 27L, 32L, 30L, 20L, 56L, 55L, 35L, 26L, 53L, 43L, 39L, 19L,
34L, 28L, 19L, 24L, 50L, 22L, 58L, 24L, 50L, 25L, 37L, 30L, 51L,
69L, 23L, 49L, 22L, 46L, 58L, 31L, 23L, 53L, 59L, 25L, 38L, 44L,
34L, 49L, 19L, 39L, 24L, 51L, 29L, 27L, 48L, 77L, 22L, 43L, 59L,
49L, 60L, 51L, 49L, 47L, 50L, 44L, 41L, 44L, 50L, 42L, 46L, 54L,
35L, 21L, 26L, 26L, 40L, 21L, 48L, 49L, 20L, 20L, 32L, 37L, 22L,
36L, 46L, 28L, 39L, 35L, 51L, 39L, 49L, 57L, 46L, 18L, 52L, 47L,
27L, 32L, 23L, 43L, 42L, 57L, 22L, 40L, 19L, 58L, 71L, 55L, 42L,
20L, 51L, 21L, 20L, 61L, 36L, 54L, 19L, 35L, 38L, 41L, 34L, 22L,
41L, 42L, 56L, 50L, 53L, 53L, 48L, 22L, 59L, 27L, 28L, 32L, 37L,
68L, 24L, 26L, 61L, 21L, 20L, 20L, 50L, 62L, 61L, 29L, 18L, 40L,
67L, 43L, 25L, 43L, 22L, 56L, 47L, 41L, 40L, 43L, 27L, 37L, 61L,
35L, 23L, 54L, 38L, 38L, 39L, 45L, 49L, 63L, 49L, 44L, 44L, 23L,
37L, 58L, 61L, 25L, 18L, 59L, 25L, 51L, 40L, 27L, 42L, 22L, 38L,
22L, 45L, 33L, 32L, 36L, 53L, 52L, 19L, 45L, 53L, 27L, 65L, 25L,
53L, 57L, 29L, 23L, 62L, 36L, 56L, 59L, 41L, 61L, 44L, 24L, 21L,
38L, 29L, 55L, 33L, 18L, 21L, 19L, 65L, 24L, 59L, 34L, 25L, 45L,
48L, 18L, 41L, 61L, 32L, 37L, 21L, 20L, 57L, 25L, 65L, 50L, 61L,
32L, 27L, 19L, 50L, 63L, 19L, 45L, 20L, 36L, 20L, 19L, 53L, 39L,
50L, 20L, 24L, 57L, 28L, 21L, 39L, 49L, 21L, 20L, 39L, 20L, 44L,
19L, 39L, 53L, 29L, 60L, 43L, 21L, 23L, 30L, 42L, 42L, 51L, 35L,
50L, 51L, 56L, 52L, 22L, 36L, 56L, 28L, 57L, 20L, 47L, 48L, 65L,
71L, 21L, 70L, 23L, 63L), Highest_Educationmx = c(4L, 5L, 3L,
2L, 3L, 2L, 3L, 1L, 3L, 1L, 7L, 3L, 2L, 3L, 3L, 2L, 6L, 2L, 3L,
6L, 3L, 2L, 2L, 7L, 2L, 1L, 2L, 3L, 6L, 3L, 5L, 3L, 5L, 6L, 2L,
1L, 5L, 2L, 5L, 1L, 1L, 3L, 2L, 3L, 1L, 7L, 5L, 4L, 7L, 3L, 1L,
1L, 6L, 3L, 3L, 2L, 4L, 6L, 5L, 4L, 2L, 6L, 1L, 3L, 4L, 2L, 1L,
5L, 5L, 3L, 1L, 5L, 3L, 3L, 1L, 4L, 2L, 3L, 5L, 3L, 1L, 4L, 2L,
1L, 2L, 7L, 2L, 5L, 3L, 2L, 6L, 1L, 1L, 3L, 4L, 1L, 5L, 1L, 3L,
4L, 2L, 7L, 2L, 4L, 4L, 7L, 4L, 6L, 3L, 1L, 2L, 1L, 5L, 5L, 1L,
5L, 2L, 7L, 3L, 4L, 2L, 4L, 2L, 4L, 2L, 2L, 4L, 1L, 2L, 1L, 2L,
6L, 1L, 2L, 5L, 2L, 2L, 5L, 1L, 6L, 5L, 2L, 1L, 2L, 1L, 1L, 3L,
2L, 4L, 3L, 2L, 3L, 1L, 5L, 5L, 7L, 1L, 3L, 3L, 2L, 1L, 3L, 4L,
5L, 1L, 1L, 3L, 3L, 3L, 5L, 3L, 6L, 4L, 3L, 1L, 3L, 5L, 7L, 1L,
3L, 4L, 5L, 3L, 3L, 1L, 1L, 1L, 7L, 3L, 1L, 4L, 3L, 3L, 5L, 1L,
4L, 5L, 4L, 2L, 5L, 3L, 1L, 1L, 5L, 4L, 7L, 5L, 2L, 2L, 5L, 3L,
1L, 1L, 2L, 3L, 5L, 3L, 7L, 5L, 1L, 5L, 3L, 1L, 1L, 1L, 1L, 7L,
5L, 7L, 3L, 1L, 5L, 7L, 6L, 3L, 7L, 2L, 2L, 3L, 1L, 2L, 1L, 5L,
5L, 2L, 4L, 1L, 1L, 2L, 1L, 4L, 7L, 3L, 2L, 5L, 3L, 2L, 4L, 2L,
1L, 7L, 5L, 2L, 2L, 2L, 3L, 4L, 1L, 2L, 5L, 2L, 3L, 3L, 1L, 3L,
2L, 3L, 5L, 1L, 3L, 1L, 5L, 4L, 5L, 4L, 5L, 5L, 5L, 1L, 3L, 3L,
1L, 3L, 6L, 3L, 4L, 3L, 3L, 5L, 3L), Censor = c(0L, 1L, 1L, 0L,
0L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 1L, 0L, 1L, 0L, 0L, 0L, 1L,
1L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 1L, 0L, 0L, 1L, 1L, 1L, 0L, 0L,
0L, 0L, 1L, 0L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 1L, 0L,
1L, 0L, 0L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 0L,
1L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L,
1L, 1L, 1L, 1L, 1L, 0L, 1L, 0L, 1L, 0L, 0L, 1L, 0L, 0L, 1L, 1L,
0L, 0L, 0L, 0L, 1L, 1L, 1L, 1L, 0L, 0L, 0L, 1L, 1L, 0L, 1L, 1L,
0L, 0L, 1L, 0L, 1L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L,
1L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 1L, 1L, 1L, 1L, 0L, 0L, 0L,
0L, 1L, 1L, 0L, 0L, 0L, 1L, 1L, 0L, 1L, 0L, 1L, 1L, 1L, 1L, 1L,
1L, 1L, 0L, 0L, 1L, 1L, 0L, 1L, 0L, 0L, 0L, 1L, 1L, 1L, 0L, 0L,
0L, 1L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 1L, 1L, 0L, 0L, 1L,
0L, 1L, 1L, 1L, 0L, 1L, 0L, 1L, 1L, 0L, 1L, 1L, 0L, 1L, 1L, 0L,
1L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 1L, 1L, 1L, 1L, 0L, 1L, 0L, 0L,
1L, 0L, 1L, 1L, 0L, 1L, 1L, 1L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 1L,
0L, 1L, 1L, 0L, 1L, 1L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 1L, 0L,
1L, 1L, 0L, 1L, 0L, 1L, 0L, 1L, 1L, 0L, 1L, 1L, 1L, 0L, 1L, 0L,
1L, 1L, 1L, 0L, 1L, 0L, 1L, 1L, 1L, 0L, 1L, 1L, 0L, 1L, 0L, 1L,
1L, 0L, 0L, 0L, 0L, 0L, 1L, 0L)), class = "data.frame",
row.names = c(NA, -300L))

library(mgcv)
library(ggplot2)

#Run the model
Model1 <- gam(SurvYear~
                (Gender)+
                s(Age, k=50)+
                s(Highest_Educationmx, k=7),
              weights=Censor, data=df, gamma=1.5, family=cox.ph())
summary(Model1)

#Build a perspective chart
vis.gam(Model1, view=c("Age","Highest_Educationmx"),
        plot.type="persp", color="gray", se=-1, theta=45, phi=25,
        xlab="Age", ylab= "Highest Education",
        ticktype="detailed", zlim=c(-5.00, 2.00))

A: Although a Generalized Additive Model (GAM) allows for non-linear relationships of an original set of predictors to outcome, the model is still linear in the coefficients of the basis functions used for the non-linear portions of the model. So the linear predictor is what it always is: the sum of products of predictors times associated regression coefficients. It's just that with a GAM the relationship between the basis functions used as "predictors" in the model and the original set of predictors isn't immediately obvious.
For a Cox model, the linear predictor is related to the log-hazard of outcome, versus the baseline condition.


text:
Q: conditional formatting with vlookup I am trying to apply conditional formatting to a spread sheet for products that have different expiry dates. Some products have a 12 month shelf life and others longer. The cells of those that are good, close to expiry (say 3 months off expiring) and have expired coloured different.
Is there a way of doing this, possibly using a vlookup table to find product then apply the conditional formatting using the dates in the spreadsheet?
see example spreadsheet
thanks
conditional formatting

redpajama_set_name: RedPajamaStackExchange


text:
#include <errno.h>
#include <fcntl.h>
#include <getopt.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/syscall.h>
#include <unistd.h>

#include "sha512.h"

// Some of the necessary structures, constants, and functions are declared in
// <linux/fs.h> or <keyutils.h> but are either incomplete (depending on your
// kernel version) or require an external library. So to simplify things, they
// are just redeclared here.

/* Begin <linux/fs.h> */
#define FS_MAX_KEY_SIZE 64

struct fscrypt_key {
  uint32_t mode;
  uint8_t raw[FS_MAX_KEY_SIZE];
  uint32_t size;
} __attribute__((packed));

#define FS_KEY_DESCRIPTOR_SIZE 8
#define FS_KEY_DESCRIPTOR_HEX_SIZE ((2 * FS_KEY_DESCRIPTOR_SIZE) + 1)

// Amount of padding
#define FS_POLICY_FLAGS_PAD_4 0x00
#define FS_POLICY_FLAGS_PAD_8 0x01
#define FS_POLICY_FLAGS_PAD_16 0x02
#define FS_POLICY_FLAGS_PAD_32 0x03
#define FS_POLICY_FLAGS_PAD_MASK 0x03

// Encryption algorithms
#define FS_ENCRYPTION_MODE_INVALID 0
#define FS_ENCRYPTION_MODE_AES_256_XTS 1
#define FS_ENCRYPTION_MODE_AES_256_GCM 2
#define FS_ENCRYPTION_MODE_AES_256_CBC 3
#define FS_ENCRYPTION_MODE_AES_256_CTS 4
#define FS_ENCRYPTION_MODE_AES_128_CBC 5
#define FS_ENCRYPTION_MODE_AES_128_CTS 6

// Policy provided via an ioctl on the topmost directory
struct fscrypt_policy {
  uint8_t version;
  uint8_t contents_encryption_mode;
  uint8_t filenames_encryption_mode;
  uint8_t flags;
  uint8_t master_key_descriptor[FS_KEY_DESCRIPTOR_SIZE];
} __attribute__((packed));

#define FS_IOC_SET_ENCRYPTION_POLICY _IOR('f', 19, struct fscrypt_policy)
#define FS_IOC_GET_ENCRYPTION_POLICY _IOW('f', 21, struct fscrypt_policy)

// Service prefixes for encryption keys
#define FS_KEY_DESC_PREFIX "fscrypt:"
#define EXT4_KEY_DESC_PREFIX "ext4:"  // For ext4 before 4.8 kernel
#define F2FS_KEY_DESC_PREFIX "f2fs:"  // For f2fs before 4.6 kernel
#define MAX_KEY_DESC_PREFIX_SIZE 8
/* End <linux/fs.h> */

/* Begin <keyutils.h> */
typedef int32_t key_serial_t;
#define KEYCTL_GET_KEYRING_ID 0     /* ask for a keyring's ID */
#define KEY_SPEC_SESSION_KEYRING -3 /* current session keyring */

key_serial_t add_key(const char *type, const char *description,
                     const void *payload, size_t plen, key_serial_t ringid) {
  return syscall(__NR_add_key, type, description, payload, plen, ringid);
}

key_serial_t keyctl_get_keyring_ID(key_serial_t id, int create) {
  return syscall(__NR_keyctl, KEYCTL_GET_KEYRING_ID, id, create);
}
/* End <keyutils.h> */

// Human-readable strings for encryption modes, indexed by the encryption mode
#define NUM_ENCRYPTION_MODES 7
const char *const mode_strings[NUM_ENCRYPTION_MODES] = {
    "INVALID",     "AES-256-XTS", "AES-256-GCM", "AES-256-CBC",
    "AES-256-CTS", "AES-128-CBC", "AES-128-CTS"};

// Valid amounts of filename padding, indexed by the padding flag
#define NUM_PADDING_VALUES 4
const int padding_values[NUM_PADDING_VALUES] = {4, 8, 16, 32};

/* util-linux style usage */
static void __attribute__((__noreturn__)) usage(FILE *out) {
  fputs(
      "\nUsage:\n"
      "  fscryptctl <command> [arguments] [options]\n"
      "\nCommands:\n"
      "  fscryptctl get_descriptor\n"
      "    Read a key from stdin, and print the hex descriptor of the key.\n"
      "  fscryptctl insert_key\n"
      "    Read a key from stdin, insert the key into the current session\n"
      "    keyring (or the user session keyring if a session keyring does not\n"
      "    exist), and print the descriptor of the key.\n"
      "  fscryptctl get_policy <file or directory>\n"
      "    Print out the encryption policy for the specified path.\n"
      "  fscryptctl set_policy <key descriptor> <directory>\n"
      "    Set up an encryption policy on the specified directory with the\n"
      "    specified key descriptor.\n"
      "\nOptions:\n"
      "    -h, --help\n"
      "        print this help screen\n"
      "    -v, --version\n"
      "        print the version of fscrypt\n"
      "    insert_key\n"
      "        --ext4\n"
      "            for use with an ext4 filesystem before kernel v4.8\n"
      "        --f2fs\n"
      "            for use with an F2FS filesystem before kernel v4.6\n"
      "    set_policy\n"
      "        --contents=<mode>\n"
      "            contents encryption mode (default: AES-256-XTS)\n"
      "        --filenames=<mode>\n"
      "            filenames encryption mode (default: AES-256-CTS)\n"
      "        --padding=<bytes>\n"
      "            bytes of zero padding for filenames (default: 32)\n"
      "\nNotes:\n"
      "  All input keys are 64 bytes long and formatted as binary.\n"
      "  All descriptors are 8 bytes and formatted as hex (16 characters).\n",
      out);

  exit(out == stderr ? EXIT_FAILURE : EXIT_SUCCESS);
}

// For getting/setting policies, our error messages might differ from the
// standard ones for certain errno values.
const char *policy_error(int errno_val) {
  // Only these errno values actually relate to filesystem encryption
  switch (errno_val) {
    case ENOTTY:
      return "your kernel is too old to support filesystem encryption, or the "
             "filesystem you are using does not support encryption";
    case EOPNOTSUPP:
      return "filesystem encryption has been disabled in the kernel config, or "
             "you need to enable encryption on your filesystem (see the README "
             "for more detailed instructions).";
    case ENODATA:
      return "file or directory not encrypted";
    case EEXIST:
      // EINVAL was returned instead of EEXIST on some filesystems before v4.11.
      // However, we do not attempt to work around this.
      return "file or directory already encrypted";
    case EINVAL:
      return "invalid encryption options provided";
    default:
      return strerror(errno_val);
  }
}

// Converts str to an encryption mode. Returns 0 (FS_ENCRYPTION_MODE_INVALID) if
// the string does not correspond to an encryption mode.
static uint8_t string_to_mode(const char *str) {
  uint8_t i;
  for (i = 1; i < NUM_ENCRYPTION_MODES; ++i) {
    if (strcmp(str, mode_strings[i]) == 0) {
      return i;
    }
  }
  return 0;
}

// Converts the encryption mode to a human-readable string. Returns "INVALID" if
// the mode is not a valid encryption mode.
static const char *mode_to_string(uint8_t mode) {
  if (mode >= NUM_ENCRYPTION_MODES) {
    mode = 0;
  }
  return mode_strings[mode];
}

// Converts an amount of padding (as a string) into the appropriate padding
// flag. Returns -1 if the flag is invalid.
static int string_to_padding_flag(const char *str) {
  int i, padding = atoi(str);
  for (i = 0; i < NUM_PADDING_VALUES; ++i) {
    if (padding == padding_values[i]) {
      return i;
    }
  }
  return -1;
}

// Takes an input key descriptor as a byte array and outputs a hex string.
static void key_descriptor_to_hex(const uint8_t bytes[FS_KEY_DESCRIPTOR_SIZE],
                                  char hex[FS_KEY_DESCRIPTOR_HEX_SIZE]) {
  int i;
  for (i = 0; i < FS_KEY_DESCRIPTOR_SIZE; ++i) {
    sprintf(hex + 2 * i, "%02x", bytes[i]);
  }
}

// Takes an input key descriptor as a hex string and outputs a bytes array.
// Returns non-zero if the provided hex string is not formatted correctly.
static int key_descriptor_to_bytes(const char *hex,
                                   uint8_t bytes[FS_KEY_DESCRIPTOR_SIZE]) {
  if (strlen(hex) != FS_KEY_DESCRIPTOR_HEX_SIZE - 1) {
    return -1;
  }

  int i, bytes_converted, chars_read;
  for (i = 0; i < FS_KEY_DESCRIPTOR_SIZE; ++i) {
    // We must read two hex characters of input into one byte of buffer.
    bytes_converted = sscanf(hex + 2 * i, "%2hhx%n", bytes + i, &chars_read);
    if (bytes_converted != 1 || chars_read != 2) {
      return -1;
    }
  }
  return 0;
}

// Reads key data from stdin into the provided data buffer. Return 0 on success.
static int read_key(uint8_t key[FS_MAX_KEY_SIZE]) {
  size_t rc = fread(key, 1, FS_MAX_KEY_SIZE, stdin);
  int end = fgetc(stdin);
  // We should read exactly FS_MAX_KEY_SIZE bytes, then hit EOF
  if (rc == FS_MAX_KEY_SIZE && end == EOF && feof(stdin)) {
    return 0;
  }

  fprintf(stderr, "error: input key must be %d bytes\n", FS_MAX_KEY_SIZE);
  return -1;
}

// The descriptor is just the first 8 bytes of a double application of SHA512
// formatted as hex (so 16 characters).
static void compute_descriptor(const uint8_t key[FS_MAX_KEY_SIZE],
                               char descriptor[FS_KEY_DESCRIPTOR_HEX_SIZE]) {
  uint8_t digest1[SHA512_DIGEST_LENGTH];
  SHA512(key, FS_MAX_KEY_SIZE, digest1);

  uint8_t digest2[SHA512_DIGEST_LENGTH];
  SHA512(digest1, SHA512_DIGEST_LENGTH, digest2);

  key_descriptor_to_hex(digest2, descriptor);
  secure_wipe(digest1, SHA512_DIGEST_LENGTH);
  secure_wipe(digest2, SHA512_DIGEST_LENGTH);
}

// Inserts the key into the current session keyring with type logon and the
// service specified by service_prefix.
static int insert_logon_key(const uint8_t key_data[FS_MAX_KEY_SIZE],
                            const char descriptor[FS_KEY_DESCRIPTOR_HEX_SIZE],
                            const char *service_prefix) {
  // We cannot add directly to KEY_SPEC_SESSION_KEYRING, as that will make a new
  // session keyring if one does not exist, rather than adding it to the user
  // session keyring.
  int keyring_id = keyctl_get_keyring_ID(KEY_SPEC_SESSION_KEYRING, 0);
  if (keyring_id < 0) {
    return -1;
  }

  char description[MAX_KEY_DESC_PREFIX_SIZE + FS_KEY_DESCRIPTOR_HEX_SIZE];
  sprintf(description, "%s%s", service_prefix, descriptor);

  struct fscrypt_key key = {.mode = 0, .size = FS_MAX_KEY_SIZE};
  memcpy(key.raw, key_data, FS_MAX_KEY_SIZE);

  int ret =
      add_key("logon", description, &key, sizeof(key), keyring_id) < 0 ? -1 : 0;

  secure_wipe(key.raw, FS_MAX_KEY_SIZE);
  return ret;
}

static int get_policy(const char *path, struct fscrypt_policy *policy) {
  // We can query the policy for a directory or a file in that directory.
  int fd = open(path, O_RDONLY);
  if (fd < 0) {
    return -1;
  }

  int ret = ioctl(fd, FS_IOC_GET_ENCRYPTION_POLICY, policy);
  close(fd);

  if (ret < 0) {
    // Kernels prior to v4.11 returned ENOENT if the file did not have an
    // encryption policy, newer kernels properly return ENODATA. This lets us
    // print the right error in policy_error regardless of kernel version.
    if (errno == ENOENT) {
      errno = ENODATA;
    }
  }

  return ret;
}

static int set_policy(const char *path, const struct fscrypt_policy *policy) {
  // Policies can only be set on directories
  int fd = open(path, O_RDONLY | O_DIRECTORY);
  if (fd < 0) {
    return -1;
  }

  int ret = ioctl(fd, FS_IOC_SET_ENCRYPTION_POLICY, policy);
  close(fd);

  return ret;
}

/* Functions for various actions, return 0 on success, non-zero on failure. */

// Get the descriptor for some key data passed via stdin. Provided key data must
// have length FS_MAX_KEY_SIZE. Output will be formatted as hex.
static int cmd_get_descriptor(int argc, char *const argv[]) {
  if (argc != 2) {
    fputs("error: unexpected arguments\n", stderr);
    return EXIT_FAILURE;
  }

  int ret = EXIT_SUCCESS;
  uint8_t key[FS_MAX_KEY_SIZE];

  if (read_key(key)) {
    ret = EXIT_FAILURE;
    goto cleanup;
  }

  char descriptor[FS_KEY_DESCRIPTOR_HEX_SIZE];
  compute_descriptor(key, descriptor);
  puts(descriptor);

cleanup:
  secure_wipe(key, FS_MAX_KEY_SIZE);
  return ret;
}

// Insert a key read from stdin into the current session keyring. This has the
// effect of unlocking files encrypted with that key.
static int cmd_insert_key(int argc, char *const argv[]) {
  // Which prefix will be used in this program, changed via command line flag.
  const char *service_prefix = FS_KEY_DESC_PREFIX;

  static const struct option insert_key_options[] = {
      {"ext4", no_argument, NULL, 'e'},
      {"f2fs", no_argument, NULL, 'f'},
      {NULL, 0, NULL, 0}};

  int ch;
  while ((ch = getopt_long(argc, argv, "", insert_key_options, NULL)) != -1) {
    switch (ch) {
      case 'e':
        service_prefix = EXT4_KEY_DESC_PREFIX;
        break;
      case 'f':
        service_prefix = F2FS_KEY_DESC_PREFIX;
        break;
      default:
        usage(stderr);
    }
  }

  // This command does not need additional arguments
  if (argc != optind + 1) {
    fputs("error: unexpected arguments\n", stderr);
    return EXIT_FAILURE;
  }

  int ret = EXIT_SUCCESS;
  uint8_t key[FS_MAX_KEY_SIZE];
  if (read_key(key)) {
    ret = EXIT_FAILURE;
    goto cleanup;
  }

  char descriptor[FS_KEY_DESCRIPTOR_HEX_SIZE];
  compute_descriptor(key, descriptor);
  if (insert_logon_key(key, descriptor, service_prefix)) {
    fprintf(stderr, "error: inserting key: %s\n", strerror(errno));
    ret = EXIT_FAILURE;
    goto cleanup;
  }
  puts(descriptor);

cleanup:
  secure_wipe(key, FS_MAX_KEY_SIZE);
  return ret;
}

// For a specified file or directory with encryption enabled, print the
// corresponding policy to stdout. Key descriptor will be formatted as hex.
static int cmd_get_policy(int argc, char *const argv[]) {
  if (argc != 3) {
    fputs("error: must specify a single file or directory\n", stderr);
    return EXIT_FAILURE;
  }
  const char *path = argv[2];

  struct fscrypt_policy policy;
  if (get_policy(path, &policy)) {
    fprintf(stderr, "error: getting policy for %s: %s\n", path,
            policy_error(errno));
    return EXIT_FAILURE;
  }

  // Pretty print the policy (includes key descriptor and flags)
  char descriptor[FS_KEY_DESCRIPTOR_HEX_SIZE];
  key_descriptor_to_hex(policy.master_key_descriptor, descriptor);
  int padding = padding_values[policy.flags & FS_POLICY_FLAGS_PAD_MASK];

  printf("Encryption policy for %s:\n", path);
  printf("\tPolicy Version: %d\n", policy.version);
  printf("\tKey Descriptor: %s\n", descriptor);
  printf("\tContents: %s\n", mode_to_string(policy.contents_encryption_mode));
  printf("\tFilenames: %s\n", mode_to_string(policy.filenames_encryption_mode));
  printf("\tPadding: %d\n", padding);

  return EXIT_SUCCESS;
}

// Apply a policy (i.e. the specified descriptor) to the specified directory.
// The policy options defaults can be overridden by command-line options.
static int cmd_set_policy(int argc, char *const argv[]) {
  // As Kernel version 4.9, the only policy field that has multiple valid
  // options is "flags", which sets the amount of zero padding on filenames.
  struct fscrypt_policy policy = {
      .version = 0,
      .contents_encryption_mode = FS_ENCRYPTION_MODE_AES_256_XTS,
      .filenames_encryption_mode = FS_ENCRYPTION_MODE_AES_256_CTS,
      // Use maximum zero-padding to leak less info about filename length
      .flags = FS_POLICY_FLAGS_PAD_32};

  // Use the command-line options to modify the policy
  static const struct option insert_key_options[] = {
      {"contents", required_argument, NULL, 'c'},
      {"filenames", required_argument, NULL, 'f'},
      {"padding", required_argument, NULL, 'p'},
      {NULL, 0, NULL, 0}};

  int ch, padding_flag;
  while ((ch = getopt_long(argc, argv, "", insert_key_options, NULL)) != -1) {
    switch (ch) {
      case 'c':
        policy.contents_encryption_mode = string_to_mode(optarg);
        if (policy.contents_encryption_mode == FS_ENCRYPTION_MODE_INVALID) {
          fprintf(stderr, "error: invalid contents mode: %s\n", optarg);
          return EXIT_FAILURE;
        }
        break;
      case 'f':
        policy.filenames_encryption_mode = string_to_mode(optarg);
        if (policy.filenames_encryption_mode == FS_ENCRYPTION_MODE_INVALID) {
          fprintf(stderr, "error: invalid filenames mode: %s\n", optarg);
          return EXIT_FAILURE;
        }
        break;
      case 'p':
        padding_flag = string_to_padding_flag(optarg);
        if (padding_flag < 0) {
          fprintf(stderr, "error: invalid padding: %s\n", optarg);
          return EXIT_FAILURE;
        }
        policy.flags = padding_flag;
        break;
      default:
        usage(stderr);
    }
  }

  // We should have exactly 2 more arguments
  if (argc != optind + 3) {
    fputs("error: must specify a key descriptor and directory\n", stderr);
    return EXIT_FAILURE;
  }
  const char *descriptor = argv[optind + 1];
  const char *path = argv[optind + 2];

  // Copy the descriptor into the policy, requires changing format.
  if (key_descriptor_to_bytes(descriptor, policy.master_key_descriptor)) {
    fprintf(stderr, "error: invalid descriptor: %s\n", descriptor);
    return EXIT_FAILURE;
  }

  if (set_policy(path, &policy)) {
    fprintf(stderr, "error: setting policy for %s: %s\n", path,
            policy_error(errno));
    return EXIT_FAILURE;
  }

  return EXIT_SUCCESS;
}

int main(int argc, char *const argv[]) {
  // Check for the help flag
  int i;
  for (i = 1; i < argc; ++i) {
    if (strcmp(argv[i], "-h") == 0 || strcmp(argv[i], "--help") == 0) {
      usage(stdout);
    }
    if (strcmp(argv[i], "-v") == 0 || strcmp(argv[i], "--version") == 0) {
      puts(VERSION);
      return EXIT_SUCCESS;
    }
  }

  if (argc < 2) {
    fputs("error: no command specified\n", stderr);
    usage(stderr);
  }
  const char *command = argv[1];

  if (strcmp(command, "get_descriptor") == 0) {
    return cmd_get_descriptor(argc, argv);
  } else if (strcmp(command, "insert_key") == 0) {
    return cmd_insert_key(argc, argv);
  } else if (strcmp(command, "get_policy") == 0) {
    return cmd_get_policy(argc, argv);
  } else if (strcmp(command, "set_policy") == 0) {
    return cmd_set_policy(argc, argv);
  }

  fprintf(stderr, "error: invalid command: %s\n", command);
  usage(stderr);
}

redpajama_set_name: RedPajamaGithub


Last week the U.S. Court of Appeals for the Federal Circuit reversed the U.S. District Court for the Northern District of California's ruling of fair use in Oracle America, Inc. v. Google LLC, and held that a verbatim and non-transformative taking in the presence of an actual or potential licensing market fatally undermined the defense. Oracle had sued Google for copyright infringement, alleging that Google had unlawfully used 37 packages of Oracle's Java application programming interface – "pre-written Java source code programs" that serve as shortcuts for various computer functions to save programming time – in its Android-powered devices. Google copied verbatim 11,500 lines of Oracle's copyrighted computer code as well as the structure, sequence, and organizing of the packages. After a second jury trial on fair use, Google prevailed on its fair use defense, and Oracle appealed after the district court rejected its post-trial motion for judgment as a matter of law.
The Federal Circuit disagreed with the district court's assessment, and analyzed each of the four fair use factors in 17 U.S.C. 107. In particular, under the first factor (nature and purpose of the use), the court held that Google's use of Oracle's code was both commercial and not "transformative" because the purpose of the software packages in Google's Android operating system was the same as the purpose of the package in Oracle's Java platform; Google did not change the expressive content or message of the code; and use of the code in smartphones as opposed to other computer hardware did not constitute "new context." As many courts do, the Federal Circuit did not pay much heed to the second factor (nature of the copyrighted work), but emphasized under the third factor (amount of the work used), that the taking at issue here was more than was defensible. For instance, there was no dispute that only 170 lines of code were needed to write in Java programming language, but Google copied 11,500 lines.
The court spent considerable time discussing the fourth factor (effect on the potential market), focusing on harm to actual markets for the copyrighted work, as well as the market for potential and derivative uses. The court noted that the record clearly showed actual market harm in that Oracle's copyrighted works had already been used in mobile devices, that Google directly competed with Oracle using Oracle's own code, and that the existence of the free Android operating system caused significant damage to Oracle's negotiating position with third parties like Amazon. The district court also had failed to consider potential market harm, including licensing Java "for smartphones with increased processing capabilities"; importantly, the court observed that just because Oracle had never built its own smartphone device was irrelevant "because potential markets include licensing others to develop derivative works." Because factors one and four weighed heavily against fair use (factor two weighed in favor, and factor three was likely against), the Federal Circuit reversed and remanded for a trial on damages.
Even in industries unrelated to computers, mobile devices, software, and source code, the court's broad pronouncement that "[t]here is nothing fair about taking a copyrighted work verbatim and using it for the same purpose and function as the original in a competing platform" is both powerful and beneficial to creators and licensors of copyrighted content. DMLA's amicus brief with the support of the coalition of Visual Artists– and one of many amicus briefs in this hotly contested case– helped explain to the court of appeals the importance of licensing markets in fair use cases in general. Ultimately DMLA supported the winning argument and contributed to the creation of appellate-level precedent that will help image licensors everywhere in responding to many infringement claims, as it turns on harm to the licensing market.
This entry was posted in Advocacy, Copyright, Digital Media Licensing Association, DMLA, Industry News, Legal and tagged amicus brief, fair use, Federal Circuit, Google, Java Code Packets, Oracle, U.S. Court of Appeals, verbatim and non-transformative on April 2, 2018 by Cathy Aron.
On Monday December 18, 2017, DMLA joined NPPA, ASMP and GAG in submitting an amicus brief before the Ninth Circuit Court of Appeals in support of VHT, a real estate photography company against Zillow, an online real estate platform on the narrow issue of the proper calculation of statutory damages based on multiple infringed images registered using a database registration consisting predominantly of photographs. The appeal by Zillow argues, among other issues, that the district court erred in awarding statutory damages to VHT based on each image infringed having independent economic value, despite being registered under a single database registration of photographs, which Zillow argues should only entitle VHT to one award of statutory damages regardless of the massive number of images infringed. The relevant question in the Zillow case hinges on what a court considers a "work" under Section 504(c)(1) of the Copyright Act, each separate image filed within the application, or the database as a whole, which would be one work.
The amicus brief supports the view that the independent value test applied by the lower court, and previously adopted by the Ninth Circuit is the correct one. The amicus brief describes the historical background of the various group registration procedures designed by the U.S. Copyright Office, to ease the administrative burden of registration of photographs which has unique challenges given the amount of images a photographer can create in a day. In particular, the database registration of photographs was developed with input from DMLA and its members (formerly PACA) to protect images distributed through online platforms, which formerly were distributed via published print catalogs. The amici argue that the form of registration should have no impact on whether the independent works covered by the registration should be considered a single work, entitled to a single statutory damage or multiple works entitled to damages for each work infringed. The outcome of the Ninth Circuit's ruling could have a major impact on the ability of image libraries and their contributing photographers to recover appropriate damages from infringers who use more than one of their photographs without permission, based on group registration, particularly those in the Ninth's Circuit's jurisdiction which includes California, Oregon, Washington state, Idaho, Montana, Nevada, Arizona, Alaska and Hawaii.
The brief, which you can find here, was filed by NPPA's Deputy General Counsel, Alicia Calzada, with support from DMLA's attorneys Nancy Wolff and Marissa Lewis of Cowan, DeBaets , Abrahams and Sheppard LLP. An amicus brief on another important issue in the case—secondary liability—was filed in support of VHT by the Copyright Alliance (link: http://copyrightalliance.org/wp-content/uploads/2017/12/Copyright-Alliance-VHT-v-Zillow-Amicus-Brief.pdf ), where DMLA is a member.
This entry was posted in Advocacy, Copyright, Copyright Office, Digital Media Licensing Association, DMLA, Legal, Press releases, U.S. Copyright Office and tagged amicus brief, asmp, GAG, Ninth Circuit Court of Appeals, NPPA, statutory damages, VHT, Zillow on December 20, 2017 by Cathy Aron.


Please scroll down the page to view all Tuning Hammers.
All of our Schaff-Hale Tuning Hammers are made in the USA, and accurately machined from the finest alloy steels. Schaff parts and tools have been the choice of 'Professional Piano Technicians' since 1884.
** There are much less expensive 'bargain tuning hammers' on the market, however, there's an old saying, "you only get what you pay for". The horror stories we hear of ruined tuning pins or tuning levers that come apart or bend, will just end up costing you more in repairs. So why not start with "The Best".....'Schaff' - You'll be glad you did.
The Schaff #7 Student & #S-8 Craftsman Levers are ideal for the Apprentice Tuner. 6" Long with a lacquered wood handle, it has a smooth tapered feel.
Schaff Tuning Levers with one-piece, non removable head.
These hammers are recommended for sweetening strings in between regular piano tunings.
Comes with 6-1/2" wood handle & nickel plated shaft with tuning tip. Overall length 12"
(The 5B tuning hammer is only used on tuning pins that are square all the way down the sides.) If your tuning pins are square on top but tapered down the sides, you require the Star Head #5A).
These Tuning Levers are similar in quality to our Schaff Hale Extension Tuning Levers except the stainless steel shaft is stationary.
Many technicians like the feel of the nylon handle as an alternative to wood. This tuning hammer has an 8" nylon handle with a chrome plated steel shaft.
Designed for the technician who wants to be as near to the tuning pin as possible.
This stainless steel hexagonal style extension lever is equipped with a 4" handle.
Overall length is 7", which can be extended to 10" long. Weighs only 11 ounces.
Comes with No. 13G head and No. 14B #2 star tip.
'The Technician's Choice' is this finest extension tuning hammer made by Schaff. Nylon handle for a better grip. The handle measures 9" with an overall length of 12". It has a chrome plated, steel hexagon shaft that allows for an extension of 5". It comes with #2 star Tip and weighs 17 ounces.
Nylon Handle 7" long with an overall length of 10" which can be extended to 15".
Weight - 14 ounces and comes with a #2 star tip.
The finest "T" tuning lever available.
Made of tough alloy steel that is nickel plated to withstand the hardest factory use. New style lever has a 5" wood handle, and an overall length of 6". - Square Tip.
This Rosewood handle is only 6" long. Overall length is 10" which can be extended to 14-1/2".
Lever weighs 16 oz and comes with the #13G Head, a #2 Star Tip and a stainless steel hexagon shaft and collar.
Schaff was the first company to introduce a Rosewood Tuning Lever to the industry.
This model includes an 8-1/2" Rosewood handle and a stainless steel collar with a non-slip hexagonal shaft. Weighs 18 ounces and is perfectly balanced.
Overall length is 11-1/2" with a possible 6-1/2" extension.
This tuning lever is modeled after the Yamaha type in design appearance.
I happened to find another tuning lever: S-18 Schaff Factory Tuning Lever - $65.25 from this site. I found this one has little longer lever (13"). I finished one round of tuning and it worked very well. The tool is solid, providing good communion with the pins. Also, the handle is big enough so that my hand had no pressure point to struggle with.
The customer service was very professional.
I recently placed an order for Balance Rail Punchings and a Schaff - Hale Piano Tuning Hammer / Tuning Lever. I received the product in a timely manner and each worked as expected. I would not only order from Piano Showcase - www.vandaking.com again, but I would also recommend them to friends and acquaintances.
It was a pleasure doing business with Piano Showcase Online. The tool I ordered worked perfectly,was of quality workmanship and the customer service excellent. I would do business with them again.
I studied piano tuning and regulation 50+ years ago at Northwestern and am finally getting back to it now that I have officially retired. They were very helpful and delivered quickly.


Two wallpapers with a view over Cape Town, city in South Africa. You see some tall buildings, houses and water. Cape Town is the second-most populous urban area in South Africa after Johannesburg.
HD wallpaper view over Cape Town and water.


our Regular Terms and the applicable Related Terms shall also apply to your use of our particular services through our site, including the submission of membership applications and the registration and attendance at our events.
We are a UK registered charity under registered charity no. 1030623, and also a company limited by guarantee registered in England and Wales under company number 2877400. Our registered office is at 16 Angel Gate, City Road, London EC1V 2PT. We are registered for VAT purposes with registration number 232 3785 13.
The information contained on our site has been prepared solely for the purpose of providing information about us and the services and resources we offer.
We may update our site and may change the content at any time. However, please note that although we make reasonable efforts to keep the content updated, any of the content on our site may be out of date at any given time. Similarly, whilst we endeavour to ensure that the information contained on our site is accurate, we do not guarantee that our site, or any content on it, will be free from errors or omissions.
If you are unsure about any of the content on our site, please do not hesitate to contact us at info@bps.ac.uk.
We will do our best to make our site available, however we cannot guarantee that our site, or any content on it, will always be available or be uninterrupted. Access to our site is permitted on a temporary basis. We may need to suspend, withdraw, discontinue or change all or any part of our site without notice. We will not be liable to you if for any reason our site is unavailable at any time or for any period.
We are the owner or the licensee of all intellectual property rights in our site, and in the material published on it. Those works are protected by copyright and treaties around the world. All such rights are reserved.
You may not use or copy any part of our site, including any material from our site, without an appropriate licence from us or our licensors, other than temporarily in the course of using our site or to keep a record of a transaction entered into using our site.
The content on our site is provided on an "as is" basis and for general information only. Although we make reasonable efforts to update the information on our site, we make no representations, warranties or guarantees, whether express or implied, that the content on our site is accurate, complete or up to date.
Your use of the information contained on our site or any downloadable item is at your own risk.
Where you choose or you are provided with, a username and password for the purpose of accessing materials through our membership portal, you must treat such login details as confidential. You must not disclose it to any third party.
If you know or suspect that anyone other than you knows your username or password, you must promptly notify us at info@bps.ac.uk.
Despite our reasonable efforts, we do not guarantee that our site will be secure or free from bugs or viruses. You should use your own virus protection software.
You must not misuse our site by knowingly introducing viruses, trojans, worms, logic bombs or other material which is malicious or technologically harmful. You must not attempt to gain unauthorised access to our site, the server on which our site is stored or any server, computer or database connected to our site. You must not attack our site via a denial-of-service attack or a distributed denial-of service attack. By breaching this provision, you may commit an offence in law, including under the Computer Misuse Act 1990. We will report any such breach to the relevant law enforcement authorities and we will co-operate with those authorities by disclosing your identity to them. In the event of such a breach, your right to use our site will cease immediately.
We ask that you do not establish a link in such a way as to suggest any form of association, approval or endorsement on our part where none exists, and you must not establish a link to our site in any website that is not owned by you.
Our site must also not be framed on any other website without our consent.
We reserve the right to reasonably withdraw linking permission without notice.
If you wish to make any use of content on our site other than that set out above, or if you have any queries in relation to linking to our site, please contact info@bps.ac.uk.
Our site may use links to other websites (including to give effect to services that we may from time to time provide). We aim to make these links clear, so that you know when you have left our site.
Please be aware that other sites are owned and run by third-parties and we have no control over the contents of those sites or resources. If you agree to purchase services offered on or through such third party websites, any agreement will be entirely between you and the third party; we will not be involved in, or have any responsibility under, those agreements.
We may from time to time provide interactive discussions on our site or provide features that allow you to upload content to our site.
You agree to comply with the following standards, and you will be responsible for any loss or damage we suffer as a result of such contribution.
Any content you upload to our site will be considered non-confidential and non-proprietary. You retain all of your ownership rights in your content, but you are required to grant us and other users of our site a perpetual, payment free, worldwide licence to use, store and copy that content and to distribute and make it available to third parties.
We also have the right to disclose your identity to any third party who may claim that any content posted or uploaded by you to our site constitutes a violation of their intellectual property rights, or their right to privacy.
that is or might be technically harmful or intrusive (including without limitation viruses, logic bombs, spyware, Trojan horses, worms and/or destructive code).
immediate, temporary or permanent withdrawal of your right to attend the event registered for.
a. To the extent permitted by law, we exclude all conditions, warranties, representations or other terms which may apply to our site or any content on it, whether express or implied.
d. If you are a consumer user, please note that we only provide our site for private use. You agree not to use our site for any commercial or business purposes, and we have no liability to you for any direct or indirect loss of profit, loss of business, business interruption, or loss of business opportunity.
e. We will not be liable for any loss or damage caused by a virus, distributed denial-of-service attack, or other technologically harmful material that may infect your computer equipment, computer programs, data or other proprietary material due to your use of our site or to your downloading of any content on it, or on any website linked to it.
f. We assume no responsibility for the content of websites or other materials linked on our site. We will not be liable for any loss or damage that may arise from your use of or linking to them.
g. You acknowledge that we are under no obligation to oversee, monitor or moderate any interactive service we provide on our site, or any content or materials posted and we expressly exclude our liability for any loss or damage arising from the use of any interactive service by a user in contravention of our content standards, whether the service is moderated or not.
BRITISH PHARMACOLOGICAL SOCIETY™, BRITISH JOURNAL OF CLINICAL PHARMACOLOGY™, BRITISH JOURNAL OF PHARMACOLOGY™ and the British Pharmacological Society logo are trade marks of the British Pharmacological Society.
In entering into any transaction using our site you do so on the basis of the terms and conditions of each transaction type which are made up of the Related Terms (for example "Membership Applications - Related Terms") and the Regular Terms set out below. Please note that more than one set of Related Terms may apply to a single transaction (e.g. where you register for a British Pharmacological Society event) and where this is the case you will be entering into a distinct contract for each transaction type.
The completion of any application for membership, registration for any event, subscription to any journals, application for any education services including bursaries and grants or any services provided via our site (individually or collectively hereinafter "Services") indicates your acceptance of these Terms and Conditions. We will treat each transaction as an offer by you to purchase such Services, subject to these Terms and Conditions. These terms and conditions do not affect your statutory rights.
The word "Terms" hereafter refers to the Related Terms together with these Regular Terms.
We reserve the right to make changes to our policies and/or these Terms at any time. You will be subject to the policies and Terms in force at the time that you use our site and/or you enter into any transaction with us, unless any change to those policies or these Terms is required to be made by law (in which case the changes will apply retrospectively). If any of these Terms are for any reason unenforceable, that provision will be deemed superseded by a valid, enforceable provision that most closely matches the intent of the original provision and the remainder of the agreement shall continue in effect, without affecting the validity and enforceability of any remaining term.
Without affecting your statutory rights, we will not be held responsible for any delay or failure to comply with our obligations under these Terms if the delay or failure arises from any cause which is beyond our reasonable control.
We reserve the right to refuse to provide any of the Services and/or refuse admission to any event or scientific meeting, at our absolute discretion.
You warrant that you possess the legal authority to enter into any transaction through our site (including instructing us to collect any payments from a payment card or billing account) and to use our site in accordance with the Terms. You agree to be financially responsible for all transactions you undertake through your use of our site as well as for use of your account or log in by others.
If we provide you with any user name and/or password, you agree that they are personal to you and you may not share them with and/or transfer them to any third party. You are solely responsible for the security of any user name and/or password and you shall immediately notify us if you are aware of any breach.
Other than liability for death or personal injury resulting from our negligence, we, our employees and/ or agents shall not be liable for any loss or damage, direct or indirect, howsoever arising out of or in connection with our relationship with you to the extent permitted by law.
In the event that the above exclusion of liability is not effective, our maximum aggregate liability (save in respect of death or personal injury resulting from our negligence) shall be the amount paid by the claimant to us in respect of the transaction out of which the claim against us arose.
All prices quoted are in pounds sterling, unless otherwise stated.
When you place an order for any Service through our site, this will be an offer by you to purchase the Service displayed in your shopping basket. Payment will be taken from your payment card when your order is completed successfully online. Acceptance of your order and the completion of our contract for the sale of the Service will take place on the confirmation (written or electronic) to you of completion of the transaction.
We use a secure, encrypted third-party payment terminal for transactions via our site and no credit or debit card details are stored, received by, made known to or held by us. If you chose to pay for a transaction directly via us (for example by providing your payment details to a member of staff over the telephone), the credit or debit card details will be input to the same third-party payment terminal and then securely destroyed.
These Terms constitute the only agreement between you and us in respect of the relevant transaction and a printed or electronic version of the Terms shall be admissible in legal or other proceedings relating to these Terms.
None of the Terms shall be enforceable under the Contracts (Rights of Third Parties) Act 1999 by a party other than you or us.
Applications for membership are received by us in good faith and are subject to a vetting process for health, safety and security purposes. As such each application and any related membership subscription will be acknowledged by us on completion of the online process and completion of the membership process will be subject to further written confirmation. In the event that for any reason an application is unsuccessful, any membership and/or journal subscriptions will be repaid by us immediately and in full.
Membership fees are based on a standard 12-month contract and membership starts on the date of purchase unless agreed otherwise by us. Membership subscriptions are renewable annually on 1st January and we reserve the right, without notice, to expunge those memberships where renewal subscriptions remain unpaid for a period of three months.
In the event that a payment is rejected by any bank or credit card company then the membership will be suspended until valid payment of the membership fee is received by us. If valid payment is not received then membership will be terminated. Any such termination may be back dated to the date when the rejected payment fell due.
Membership cannot be cancelled during a period which has already been paid for.
Registrations will be subject to an approval process and, on completion of online registration, a confirmation will be automatically sent by email which should be printed and carefully retained as you will need this to gain admittance to the meeting. If for any reason a registration is declined, any online payment made will be refunded immediately and in full. This process is necessary for health, safety and security reasons and we are most grateful to all non-member registrants for their understanding.
Where specified in the information about a meeting or other event, a late registration fee will be applied for late registrations.
a legible print-out of your registration confirmation. Failure to bring a print out with you may delay the admission process.
We reserve the right to charge the Full Registration Fee to any person not able to validate their registration in the above manner.
No event registration made through our site may be resold or transferred without our prior written consent. If any person attempts to enter an event with a registration that has been resold or transferred without our written consent, we may refuse entry and no compensation will be payable to that person by us.
By attending our events and meetings, delegates consent to the taking of photographs and accept that these images may appear on our site and other British Pharmacological Society publicity materials.
To contact us, please email info@bps.ac.uk.


text:
Q: Join two Spark DStreams with complex nested structure I have implemented a Spark custom receiver to receive DStreams from http/REST as follows
val mem1Total:ReceiverInputDStream[String] = ssc.receiverStream(new CustomReceiver("httpURL1"))
val dstreamMem1:DStream[String] = mem1Total.window(Durations.seconds(30), Durations.seconds(10))
val mem2Total:ReceiverInputDStream[String] = ssc.receiverStream(new CustomReceiver("httpURL2"))
val dstreamMem2:DStream[String] = mem2Total.window(Durations.seconds(30), Durations.seconds(10))

Each stream has the following schema
val schema = StructType(Seq(
            StructField("status", StringType),
            StructField("data", StructType(Seq(
                StructField("resultType", StringType),
                StructField("result", ArrayType(StructType(Array(
                    StructField("metric", StructType(Seq(StructField("application", StringType),
                                                         StructField("component", StringType),
                                                         StructField("instance", StringType)))),
                    StructField("value", ArrayType(StringType))
                ))))
             )
         ))))

Here is what I how far i could go to extract the features that I need from dstreamMem1.
  dstreamMem1.foreachRDD { rdd =>
  import sparkSession.implicits._
  val df = rdd.toDS()
                    .selectExpr("cast (value as string) as myData")
                    .select(from_json($"myData", schema).as("myDataEvent"))
                    .select($"myDataEvent.data.*")
                    .select(explode($"result").as("flat"))
                    .select($"flat.metric.*", $"flat.value".getItem(0).as("value1"), $"flat.value".getItem(1).as("value2"))
                }

However I am cant figure out how to join dstreamMem1 with dstreamMem2 while also dealing with the complex structure. I can do a union operation on dstreamMem1  and dstreamMem2. But that wont work in my case because the "value" fields represent different things on each stream.  Any ideas please?
Edit#1
Based on the following resources
    How to create a custom streaming data source?
https://github.com/apache/spark/pull/21145
https://github.com/hienluu/structured-streaming-sources/tree/master/streaming-sources/src/main/scala/org/structured_streaming_sources/twitter
I have been able to create the following class to implement the following
        class SSPStreamMicroBatchReader(options: DataSourceOptions) extends MicroBatchReader with Logging {

          private val httpURL = options.get(SSPStreamingSource.HTTP_URL).orElse("") //.toString()
          private val numPartitions = options.get(SSPStreamingSource.NUM_PARTITIONS).orElse("5").toInt
          private val queueSize = options.get(SSPStreamingSource.QUEUE_SIZE).orElse("512").toInt
          private val debugLevel = options.get(SSPStreamingSource.DEBUG_LEVEL).orElse("debug").toLowerCase

          private val NO_DATA_OFFSET = SSPOffset(-1)
          private var startOffset: SSPOffset = new SSPOffset(-1)
          private var endOffset: SSPOffset = new SSPOffset(-1)
          private var currentOffset: SSPOffset = new SSPOffset(-1)
          private var lastReturnedOffset: SSPOffset = new SSPOffset(-2)
          private var lastOffsetCommitted : SSPOffset = new SSPOffset(-1)

          private var incomingEventCounter = 0;
          private var stopped:Boolean = false

          private var acsURLConn:HttpURLConnection = null
          private var worker:Thread = null

          private val sspList:ListBuffer[StreamingQueryStatus] = new ListBuffer[StreamingQueryStatus]()
          private var sspQueue:BlockingQueue[StreamingQueryStatus] = null

          initialize()

          private def initialize(): Unit = synchronized {
            log.warn(s"Inside initialize ....")
            sspQueue = new ArrayBlockingQueue(queueSize)

            new Thread("Socket Receiver") { log.warn(s"Inside thread ....")
              override def run() {
                log.warn(s"Inside run ....")
                receive() }
            }.start()
          }

          private def receive(): Unit = {
            log.warn(s"Inside recieve() ....")
            var userInput: String = null

              acsURLConn = new AccessACS(httpURL).getACSConnection();

              // Until stopped or connection broken continue reading
              val reader = new BufferedReader(
                new InputStreamReader(acsURLConn.getInputStream(), java.nio.charset.StandardCharsets.UTF_8))

              userInput = reader.readLine()

              while(!stopped) {
                  // poll tweets from queue
                  val tweet:StreamingQueryStatus = sspQueue.poll(100, TimeUnit.MILLISECONDS)

                  if (tweet != null) {
                    sspList.append(tweet);
                    currentOffset = currentOffset + 1
                    incomingEventCounter = incomingEventCounter + 1;
                  }
                }
              reader.close()

          }

          override def planInputPartitions(): java.util.List[InputPartition[org.apache.spark.sql.catalyst.InternalRow]] = {
            synchronized {
              log.warn(s"Inside planInputPartitions ....")

              //initialize()

              val startOrdinal = startOffset.offset.toInt + 1
              val endOrdinal = endOffset.offset.toInt + 1

              internalLog(s"createDataReaderFactories: sOrd: $startOrdinal, eOrd: $endOrdinal, " +
                s"lastOffsetCommitted: $lastOffsetCommitted")

              val newBlocks = synchronized {
                val sliceStart = startOrdinal - lastOffsetCommitted.offset.toInt - 1
                val sliceEnd = endOrdinal - lastOffsetCommitted.offset.toInt - 1
                assert(sliceStart <= sliceEnd, s"sliceStart: $sliceStart sliceEnd: $sliceEnd")
                sspList.slice(sliceStart, sliceEnd)
              }

              newBlocks.grouped(numPartitions).map { block =>
                                                        new SSPStreamBatchTask(block).asInstanceOf[InputPartition[org.apache.spark.sql.catalyst.InternalRow]]
                                                   }.toList.asJava
            }
          }

          override def setOffsetRange(start: Optional[Offset],
                                      end: Optional[Offset]): Unit = {

            if (start.isPresent && start.get().asInstanceOf[SSPOffset].offset != currentOffset.offset) {
              internalLog(s"setOffsetRange: start: $start, end: $end currentOffset: $currentOffset")
            }

            this.startOffset = start.orElse(NO_DATA_OFFSET).asInstanceOf[SSPOffset]
            this.endOffset = end.orElse(currentOffset).asInstanceOf[SSPOffset]
          }

          override def getStartOffset(): Offset = {
            internalLog("getStartOffset was called")
            if (startOffset.offset == -1) {
              throw new IllegalStateException("startOffset is -1")
            }
            startOffset
          }

          override def getEndOffset(): Offset = {
            if (endOffset.offset == -1) {
              currentOffset
            } else {

              if (lastReturnedOffset.offset < endOffset.offset) {
                internalLog(s"** getEndOffset => $endOffset)")
                lastReturnedOffset = endOffset
              }

              endOffset
            }

          }

          override def commit(end: Offset): Unit = {
            internalLog(s"** commit($end) lastOffsetCommitted: $lastOffsetCommitted")

            val newOffset = SSPOffset.convert(end).getOrElse(
              sys.error(s"SSPStreamMicroBatchReader.commit() received an offset ($end) that did not " +
                s"originate with an instance of this class")
            )

            val offsetDiff = (newOffset.offset - lastOffsetCommitted.offset).toInt

            if (offsetDiff < 0) {
              sys.error(s"Offsets committed out of order: $lastOffsetCommitted followed by $end")
            }

            sspList.trimStart(offsetDiff)
            lastOffsetCommitted = newOffset
          }

          override def stop(): Unit = {
            log.warn(s"There is a total of $incomingEventCounter events that came in")
            stopped = true
            if (acsURLConn != null) {
              try {
                //acsURLConn.disconnect()
              } catch {
                case e: IOException =>
              }
            }
          }

          override def deserializeOffset(json: String): Offset = {
            SSPOffset(json.toLong)
          }
          override def readSchema(): StructType = {
            SSPStreamingSource.SCHEMA
          }

          private def internalLog(msg:String): Unit = {
            debugLevel match {
              case "warn" => log.warn(msg)
              case "info" => log.info(msg)
              case "debug" => log.debug(msg)
              case _ =>
            }
          }
        }

        object SSPStreamingSource {

          val HTTP_URL = "httpURL"
          val DEBUG_LEVEL = "debugLevel"
          val NUM_PARTITIONS = "numPartitions"
          val QUEUE_SIZE = "queueSize"

           val SCHEMA = StructType(Seq(
                            StructField("status", StringType),
                            StructField("data", StructType(Seq(
                                StructField("resultType", StringType),
                                StructField("result", ArrayType(StructType(Array(
                                                         StructField("application", StringType),
                                                                         StructField("component", StringType),
                                                                       StructField("instance", StringType)))),
                                    StructField("value", ArrayType(StringType))
                                ))))
                             )
                         ))))
        }

        class SSPStreamBatchTask(sspList:ListBuffer[StreamingQueryStatus]) extends InputPartition[Row] {
          override def createPartitionReader(): InputPartitionReader[Row] = new SSPStreamBatchReader(sspList)
        }

        class SSPStreamBatchReader(sspList:ListBuffer[StreamingQueryStatus]) extends InputPartitionReader[Row] {
          private var currentIdx = -1

          override def next(): Boolean = {
            // Return true as long as the new index is in the seq.
            currentIdx += 1
            currentIdx < sspList.size
          }

          override def get(): Row = {
            val tweet = sspList(currentIdx)
            Row(tweet.json)
          }

          override def close(): Unit = {}
        }

Further this class is used as follows
    val a = sparkSession.readStream
            .format(providerClassName)
            .option(SSPStreamingSource.HTTP_URL, httpMemTotal)
            .load()
    a.printSchema()

    a.writeStream
            .outputMode(OutputMode.Append())
            .option("checkpointLocation", "/home/localCheckpoint1") //local
            .start("/home/sparkoutput/aa00a01")

Here is the error. Im yet to crack this :(
    18/11/29 13:33:28 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
    18/11/29 13:33:28 WARN SSPStreamingSource: Inside createMicroBatchReader() ....
    18/11/29 13:33:28 WARN SSPStreamMicroBatchReader: Inside initialize ....
    18/11/29 13:33:28 WARN SSPStreamMicroBatchReader: Inside thread ....
    18/11/29 13:33:28 WARN SSPStreamMicroBatchReader: There is a total of 0 events that came in
    18/11/29 13:33:28 WARN SSPStreamMicroBatchReader: Inside run ....
    18/11/29 13:33:28 WARN SSPStreamMicroBatchReader: Inside recieve() ....
    root
     |-- status: string (nullable = true)
     |-- data: struct (nullable = true)
     |    |-- resultType: string (nullable = true)
     |    |-- result: array (nullable = true)
     |    |    |-- element: struct (containsNull = true)
     |    |    |    |-- metric: struct (nullable = true)
     |    |    |    |    |-- application: string (nullable = true)
     |    |    |    |    |-- component: string (nullable = true)
     |    |    |    |    |-- instance: string (nullable = true)
     |    |    |    |-- value: array (nullable = true)
     |    |    |    |    |-- element: string (containsNull = true)

    18/11/29 13:33:30 INFO MicroBatchExecution: Starting [id = f15252df-96d8-45b4-a6db-83fd4c7aed71, runId = 65a6dc28-5eb4-468a-80c3-f547504689d7]. Use file:///home/localCheckpoint1 to store the query checkpoint.
    18/11/29 13:33:30 WARN SSPStreamingSource: Inside createMicroBatchReader() ....
    18/11/29 13:33:30 WARN SSPStreamMicroBatchReader: Inside initialize ....
    18/11/29 13:33:30 ERROR StreamingContext: Error starting the context, marking it as stopped
    java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute
      at scala.Predef$.require(Predef.scala:224)
      at org.apache.spark.streaming.DStreamGraph.validate(DStreamGraph.scala:168)
      at org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:513)
      at org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:573)
      at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:572)
      at myproject.spark.predictive_monitoring.predictmyproject$.run(predictmyproject.scala:99)
      at myproject.spark.predictive_monitoring.predictmyproject$.main(predictmyproject.scala:31)
      at myproject.spark.predictive_monitoring.predictmyproject.main(predictmyproject.scala)
    Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute
      at scala.Predef$.require(Predef.scala:224)
      at org.apache.spark.streaming.DStreamGraph.validate(DStreamGraph.scala:168)
      at org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:513)
      at org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:573)
      at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:572)
      at myproject.spark.predictive_monitoring.predictmyproject$.run(predictmyproject.scala:99)
      at myproject.spark.predictive_monitoring.predictmyproject$.main(predictmyproject.scala:31)
      at myproject.spark.predictive_monitoring.predictmyproject.main(predictmyproject.scala)
    18/11/29 13:33:30 INFO SparkContext: Invoking stop() from shutdown hook
    18/11/29 13:33:30 WARN SSPStreamMicroBatchReader: Inside thread ....
    18/11/29 13:33:30 WARN SSPStreamMicroBatchReader: Inside run ....
    18/11/29 13:33:30 WARN SSPStreamMicroBatchReader: Inside recieve() ....
    18/11/29 13:33:30 INFO MicroBatchExecution: Using MicroBatchReader [myproject.spark.predictive_monitoring.SSPStreamMicroBatchReader@74cc1ddc] from DataSourceV2 named 'myproject.spark.predictive_monitoring.SSPStreamingSource' [myproject.spark.predictive_monitoring.SSPStreamingSource@7e503c3]
    18/11/29 13:33:30 INFO SparkUI: Stopped Spark web UI at http://172.16.221.232:4040
    18/11/29 13:33:30 ERROR MicroBatchExecution: Query [id = f15252df-96d8-45b4-a6db-83fd4c7aed71, runId = 65a6dc28-5eb4-468a-80c3-f547504689d7] terminated with error
    java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
    This stopped SparkContext was created at:

    org.apache.spark.SparkContext.<init>(SparkContext.scala:76)
    org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:838)
    org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:85)
    myproject.spark.predictive_monitoring.predictmyproject$.run(predictmyproject.scala:37)
    myproject.spark.predictive_monitoring.predictmyproject$.main(predictmyproject.scala:31)
    myproject.spark.predictive_monitoring.predictmyproject.main(predictmyproject.scala)

    The currently active SparkContext was created at:

    org.apache.spark.SparkContext.<init>(SparkContext.scala:76)
    org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:838)
    org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:85)
    myproject.spark.predictive_monitoring.predictmyproject$.run(predictmyproject.scala:37)
    myproject.spark.predictive_monitoring.predictmyproject$.main(predictmyproject.scala:31)
    myproject.spark.predictive_monitoring.predictmyproject.main(predictmyproject.scala)

      at org.apache.spark.SparkContext.assertNotStopped(SparkContext.scala:100)
      at org.apache.spark.sql.SparkSession.<init>(SparkSession.scala:91)
      at org.apache.spark.sql.SparkSession.cloneSession(SparkSession.scala:256)
      at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:268)
      at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189)
    18/11/29 13:33:30 WARN SSPStreamMicroBatchReader: There is a total of 0 events that came in
    18/11/29 13:33:30 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
    18/11/29 13:33:30 INFO MemoryStore: MemoryStore cleared
    18/11/29 13:33:30 INFO BlockManager: BlockManager stopped
    18/11/29 13:33:30 INFO BlockManagerMaster: BlockManagerMaster stopped
    18/11/29 13:33:30 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
    18/11/29 13:33:31 INFO SparkContext: Successfully stopped SparkContext


text:
4.2.3 Mahout kmeans
Cloud Computing Applications, Part 2: Big Data and Applications in the Cloud
Иллинойсский университет в Урбане-Шампейне
4.2 (оценок: 206) | Зарегистрировано учащихся: 16K
Курс 4 из 6 — Облачные вычисления Специализация
Welcome to the Cloud Computing Applications course, the second part of a two-course series designed to give you a comprehensive view on the world of Cloud Computing and Big Data! In this second course we continue Cloud Computing Applications by exploring how the Cloud opens up data analytics of huge volumes of data that are static or streamed at high velocity and represent an enormous variety of information. Cloud applications and data analytics represent a disruptive change in the ways that society is informed by, and uses information. We start the first week by introducing some major systems for data analysis including Spark and the major frameworks and distributions of analytics applications including Hortonworks, Cloudera, and MapR. By the middle of week one we introduce the HDFS distributed and robust file system that is used in many applications like Hadoop and finish week one by exploring the powerful MapReduce programming model and how distributed operating systems like YARN and Mesos support a flexible and scalable environment for Big Data analytics. In week two, our course introduces large scale data storage and the difficulties and problems of consensus in enormous stores that use quantities of processors, memories and disks. We discuss eventual consistency, ACID, and BASE and the consensus algorithms used in data centers including Paxos and Zookeeper. Our course presents Distributed Key-Value Stores and in memory databases like Redis used in data centers for performance. Next we present NOSQL Databases. We visit HBase, the scalable, low latency database that supports database operations in applications that use Hadoop. Then again we show how Spark SQL can program SQL queries on huge data. We finish up week two with a presentation on Distributed Publish/Subscribe systems using Kafka, a distributed log messaging system that is finding wide use in connecting Big Data and streaming applications together to form complex systems. Week three moves to fast data real-time streaming and introduces Storm technology that is used widely in industries such as Yahoo. We continue with Spark Streaming, Lambda and Kappa architectures, and a presentation of the Streaming Ecosystem. Week four focuses on Graph Processing, Machine Learning, and Deep Learning. We introduce the ideas of graph processing and present Pregel, Giraph, and Spark GraphX. Then we move to machine learning with examples from Mahout and Spark. Kmeans, Naive Bayes, and fpm are given as examples. Spark ML and Mllib continue the theme of programmability and application construction. The last topic we cover in week four introduces Deep Learning technologies including Theano, Tensor Flow, CNTK, MXnet, and Caffe on Spark.
Получаемые навыки
Graphs, Distributed Computing, Big Data, Machine Learning
Very good introduction of application concepts of cloud data computing. Thank You!
good things to learn about real world big problems
Module 4: Graph Processing and Machine Learning
In this module, we discuss the applications of Big Data. In particular, we focus on two topics: graph processing, where massive graphs (such as the web graph) are processed for information, and machine learning, where massive amounts of data are used to train models such as clustering algorithms and frequent pattern mining. We also introduce you to deep learning, where large data sets are used to train neural networks with effective results.
4.2.1 Big Data Machine Learning Introduction13:01
4.2.2 Mahout: Introduction8:58
4.2.3 Mahout kmeans5:02
4.2.4 Mahout: Naïve Bayes9:27
4.2.5 Mahout: fpm6:42
4.2.6 Spark Naïve Bayes2:38
4.2.7 Spark fpm2:49
4.2.8 Spark ML/MLlib11:41
4.2.9 Introduction to Deep Learning20:20
4.2.10 Deep Neural Network Systems17:33
Reza Farivar
Data Engineering Manager at Capital One, Adjunct Research Assistant Professor of Computer Science
Roy H. Campbell
Выбор языкаАнглийскийКорейский
[NOISE] So
we're going to look at some examples of Mahout.
We'll look at Clustering, Classification, Frequency Pattern Mining.
Clustering's the first one we'll do,
and we're going to choose K-means as the sort of scheme to do that.
And let's have a look at how K-means works.
So if we got a bunch of points distributed, xy points, what
we're going to do is to choose some of those to represent our total population.
And then we're going to see if they fit into clusters.
If we can find the right clusters, we could, perhaps, go on and
classify all the other points and see how that works, as well.
So what we're going to have as an initial measure,
initial sort of means, we're going to select some items.
We're going to calculate roughly what sort of groups we have in this system,
how many groups.
We could say one, two, three, four, five.
And then calculate for each of those
roughly selected groups, what their center is.
And once we've calculated their center,
we need to calculate the centroid of this group, or that group, or that group.
We then use that to partition the group and see if that changed anything.
If that's changed anything,
we keep repeating that process until we come up with a very, very steady
set of clusters that don't change as we keep repeating the algorithm.
Once it converges, we know we've got a pretty good answer, and
that's the one we're going to use.
So what it looks like, we select K points at random.
And then we're going to take for each data point,
assign it to the closest center, so we we form K clusters.
And then for each cluster, recompute the centers.
In the case of 2D points,
we can take the X average over all x-axis points in the cluster.
We're going to take the Y average over all the y-axis average points in the cluster.
And, finally, if the new centers are different from the old centers,
then we go back and repeat the whole process.
So looking at the input, all these points,
we would input some centroids, K points, small number of them that we think there
will be that number of clusters in this system, and where they might roughly be.
So here's one, there's one and there's one.
And then we're going to map each of those K-centroids one block from one dataset,
assign each point to the closest centroid, and
then output what that central point is.
So, we've got a distributed algorithm that can be actually run over a large
number of machines, and done very quickly.
On the reduced side, we're going to get all the points for a given centroid,
recompute a new centroid for this cluster, and now put the whole for the centroid.
So for the whole process,
we're going to keep iterating, comparing the old and new set of K-centroids.
If they're similar we're going to stop.
Otherwise, what we're going to do is keep going.
We do need to make sure that we don't run forever, so we'll perhaps have a max
iterations, and stop at the end of that if we don't actually converge.
And it is possible that you could have
distributions there that don't really converge.
And so that is a sort of necessary protection for keeping your money,
for keeping your computation from not just being used.
We can add to that various optimizations.
We can add combiners to take values and assign them together, or
we can use a single reducer.
In terms of the single combiner, or use of combiners, it's similar to the reducer.
What it does is computer for each centroid the local sums (and counts)of
the assigned points, and sends that to the reducer.
In terms of a single reducer, the amount of data being sent through reducers is
very small, so a single reducer can tell whether any of its centers has changed or
not, and it can create a single output file.
So that single reducer can sort of reduce the amount of
computation and communication that needs to go on.

redpajama_set_name: RedPajamaCommonCrawl


Ready, Set, Go Crush them Goals Mama!
Hi everyone! There will be no new post this week because I am busy on working on some new content and updating the site a little bit. Thank you all for following me on this journey and I will be back next week with some brand new content.
P.S. There will be a recipe next week, so keep your eyes peeled for the post!


In case you aren't able to devote your time and vitality to a full organic backyard or just do not have the area for one, use a container as an alternative. Look for plants which are small and nicely suited to containers. Whiskey barrels are great for container gardens as a result of they've loads of room for roots to develop and can be utilized for multiple varieties of crops. The most effective ways to achieve success at natural gardening is to plan early. The very best laid out plans for an natural backyard, all the time make for essentially the most successful garden. Plan what you will plant early on and make sure to have back-up vegetables able to plant when brief-lived plants, like lettuce and spinach, are executed for the year.
Do not harm your native critters. Some animals can naturally hold the bug inhabitants down; one such instance of a very good pest-predator is the bat. Bats are effectively-identified for being bug shoppers. Since your garden might sometimes look like a tasty treat to these tiny critters, having bats round can help scale back their population naturally, without the usage of dangerous pesticides.
Work properly in your garden. One of many greatest time wasters isn't preserving track of your instruments and having to search for them every time you need them. By keeping your tools in a sure space, they are going to always be ready each time you might be. In the event you use a number of instruments, consider using a software belt or even just a few pants that have lots of pockets. You have to to rotate the vegetation regularly when you have got an indoor natural backyard. Plants have to get gentle from all directions in order to develop properly. If they are not rotated, vegetation will bend towards a light-weight supply, which might truly trigger them to provide much less vegatables and fruits, than they'd have if that they had been rotated.
In case your backyard soil is not conducive to an natural backyard, strive putting in a raised mattress. Within the raised bed, you can create your individual mixture of soil and compost to achieve the perfect soil for raising your crops. Just be sure the mattress is at the very least 16 inches excessive so that roots have room to flourish. Some householders put an excessive amount of care and detail into their yards. From ponds and stone walkways to rose beds and gazebos, folks take landscaping severely. Something that will look good in your yard this 12 months is an organic garden. When you're undecided about the way to backyard organically, here are some ideas.
Are you aware methods to start your personal natural backyard now? Can you now discover a place to begin with it? Have you learnt what will work to your seeds? If you can now provide a solution to these questions, then you've got read and understood the earlier suggestions and are able to develop your individual natural garden. You will have finally understood how an organic garden may save you money on groceries. What great timing. The following article is full of suggestions you should use to develop amazing natural plants without spending some huge cash.
Consider constructing raised beds. For those who build your own raised beds, you can select the proper measurement on your backyard, and you may fill it with a sort of soil that's appropriate for what you plan to grow. Since there will be little soil compaction, there will probably be more oxygen within the soil, and water drainage might be significantly better. The soil in a raised mattress warms up a lot earlier within the spring, increasing the growing season.
Use a raised backyard mattress when planting your plants. Not only does it provide a minor protection towards the frequent vegetable pests, raised backyard beds are additionally hotter throughout the spring. The planter turns into warmer as a result of it isn't surrounded by several inches of isolating floor-soil. The hotter climate will result you having the ability to plant earlier. Gardening is a incredible stress relieving activity that many individuals swear by. Making your personal natural vegetable garden is a good way to enhance your life-style, while additionally producing your personal meals, saving you money and offering you with high quality produce proper out of your yard. Read on to learn how to have extra success at it.
Ensure that you've earthworms in your soil. Earthworms are vital to good natural gardening, as a result of they aerate the soil. Also, the by-merchandise of earthworm digestion are literally nice plant food. Earthworms encourage soil bacteria that provide needed vitamins to your crops whereas competing with harmful insect pests, lowering the need for chemical pesticides and fertilizers.
Soaker hoses not solely preserve water, but also direct the water precisely the place it must go, into the dust, rather than on the leaves and into the air.